In a study by a team at the intersection of information and communication sciences and disorders, researchers worked to design an interactive, online professional development system for academic librarians to better serve students with autism spectrum disorder (ASD). In creating this program, it was imperative to have stakeholder input and support; recruiting members of this population, students with ASD, was critical. Amazon’s Mechanical Turk and online discussion forums, including Reddit, were used for recruitment for an online survey. While there was some overlap in results, there were also marked differences in responses based on online sampling frame. This paper details the online methods used for recruiting members of this community, and compares and contrasts success rates, challenges, and numbers associated with each method.Contents
Introduction
Related work
Methodology
Results
Discussion
Conclusions
In a study which sought to design an evidence-based professional development system for academic librarians to better understand and serve college students on the autism spectrum, information about these students about their library usage was necessary. This paper describes the process of recruiting students with autism spectrum disorder (ASD) and compares results from two online recruitment methods: 1) through online discussion forums with a specific ASD user base; and, 2) through Amazon’s Mechanical Turk (MTurk).
To provide academic librarians with the best information about working with and providing services for students on the spectrum, the voices of those students themselves needed to be included. In this context, recruiting those students and gathering data from them about their experiences in the library, with library materials, and with library staff was critical. Working with an advisory board of both academics, practitioners, and individuals on the spectrum themselves, calls for participation, along with a survey, were distributed through multiple methods. In this process, researchers found participation to be less active than anticipated, and thus began the current research focus, in conjunction with the study goals as a whole, to examine the population of college students with ASD as a hard-to-reach population.
This paper uses the term “hard-to-reach” population while also drawing from the term “hidden.” As described by Sydor, a hard-to-reach population is one in which the “populations are difficult for researchers to access” and a hidden population is one “with no defined limits” [1].
To address the topic of recruiting participants that are representative of the target population, this paper explores the following research questions:
RQ1: What online methods are more successful for recruiting college students with autism as research participants?
RQ2: How does data gathered from college students on the autism spectrum compare when subjects are recruited from two distinct online sources?
Recruiting research participants that are representative of a population is important for a successful study. While more attention is now being given to college students with ASD, until recently not much was known about members of this population beyond that which was understood by autistics themselves. It is only with appropriate recruiting efforts that more can be learned and applied toward improving their experiences in a neurotypically-driven environment. This paper reviews the literature that shapes the study, examining college students on the autism spectrum, as well as methodologies used for recruiting hard-to-reach populations.
College students with ASD and libraries
College students with autism use the library just as other students do, but organizational barriers exist that can hinder access and use. The accrediting body for librarianship, the American Library Association (ALA), claims “access” as a core value (ALA, 2019), but recent work indicates that libraries are not always accessible to students on the autism spectrum, who may experience sensory barriers in entering and using the space (Anderson, 2018).
Involving participants on the spectrum
Previous work suggests that more must be done to ensure that concepts of access translate to practices in the field; among all suggestions are those to “engage people with disabilities in our planning, decision-making, and service-design activities” and “publish perspectives of people with disabilities and disability studies scholars in the professional literature” [2]. It is not enough to enact change in a library based on what problem needs to be “solved;” instead, “in order to understand how users with disabilities access and experience library spaces, services, and resources, we need to ask them” [3]. Emily Lawrence, a self-advocate and librarian, describes the importance of collaboration with autistics and autistic-run organizations as a logical practice for librarians working to provide better services and programs:
Working directly with Autistics ensures that librarians are not relying on health care professionals, caregivers, or anyone else to speak for Autistics in the library. Librarians generally agree that they should work directly with users to determine their information needs. Applying this thinking to Autistics is really just a reasonable extension of current practice. [4] Researchers are also pushed to do a better job in amplifying the voices of people with disabilities through publication, as much literature in library and information studies (LIS) does not feature perspectives as told by librarians or library users with disabilities themselves (Kumbier and Starkey, 2016; Lawrence, 2013).
Work at the intersection of critical disability studies and librarianship indicates that more needs to be done to amplify the voices of library users and stakeholders with disabilities, involving their participation more comprehensively in research and professional activities supposedly designed for their benefit (Kumbier and Starkey, 2016; Lawrence, 2013). “Nothing about us without us” is a commonly used phrase in disability self-advocacy, and has been adopted within the mission for the Autistic Self Advocacy Network (ASAN, 2019). In accordance with this viewpoint, college students on the autism spectrum should be involved when designing services to support their preferences and needs.
College students with ASD as a hard-to-reach population
Hard-to-research or hard-to-reach have been described traditionally in medical and public health literature as those who are “socially disadvantaged” or of “lower socioeconomical status” (Bonevski, et al., 2014), while hidden populations can be described as those in which there are no defined limits or parameters (Faugier and Sargeant, 1997; Heckathorn, 1997). Being on the autism spectrum does not in and of itself make a person part of a hard-to-reach population, nor can it or should it be said that members of this population are “socially disadvantaged.” Indeed, many self-advocacy groups exist and their members are highly visible, working actively to advocate for and enact positive change. However, it is true that this population has no defined parameters; while estimates are that between .7 to 1.9 percent of undergraduates meet autism diagnostic criteria, exact numbers of college students who are on the autism spectrum are unknown, as many are undiagnosed or choose not to register for support through their disability services office (White, et al., 2011). However, issues of disclosure, stigma, and distrust of a neurotypical system might dissuade their interest in research participation.
While many on the autism spectrum claim their diagnosis proudly and view ASD as a neurodiversity, not a disorder, others are hesitant to disclose and identify as neurodivergent (Davidson and Henderson, 2010; Huws and Jones, 2008). College students in general, not just those who are neurodivergent, have a “basic human need” to fit in with their peers, and might be hesitant to proclaim anything that might make them seem different [5]. Furthermore, a “sense of belonging is a key factor for students who have been historically underrepresented in higher education” [6].
For those students who wish to keep their ASD status private, agreeing to participate in a study specifically about ASD could present a barrier. In what might add to the challenge for students on the spectrum is a shift in the language used. In 2013, the DSM-V removed the diagnosis of Asperger’s syndrome and instead incorporated the term into the broader diagnosis of ASD, or autism. For some individuals already given a diagnosis of Asperger’s syndrome, this led to a crisis of identity (Giles, 2014). What’s more, college students surveyed hold stigmatizing views toward their peers when told they had autism as opposed to Asperger’s syndrome (Butler and Gillis, 2011). If a student feels that there is a negativity associated with the term autism, it is far less likely that they will identify as member of that population. As described in one study, when compared with those who face other personal or sociodemographic barriers, “equally difficult ... to recruit for participation in research studies are those in which the target group has become stigmatized, either by personal circumstance or by association” [7].
Adults on the spectrum might also not want to participate in research if they feel uncomfortable with the manner in which it is presented. A person with ASD will be deterred if he or she feels that the research will result in tokenism, or if researchers:
[D]o not clearly demonstrate in their research design and the conduct of their studies, from recruitment through to data collection and reporting, in a way that they both understand and that is sensitive to particular and individual cognitive and communication styles, behaviours and sensory needs of people on the autism spectrum. [8] Potential research participants on the autism spectrum want their voices to be heard and valued, and they want to be seen as equal partners in the studies in which they participate (Haas, et al., 2016). In this context, if college student on the spectrum does not feel that a research study is appropriately designed to best present their experiences, and if they do not feel like equal partners in the process, it is less likely they will participate.
Strategies for recruiting hard-to-reach populations
Incentives. One commonly used tactic for participant recruiting is providing incentives, typically through financial compensation. However, one study about recruiting adults with what they describe as high functioning autism or Asperger’s syndrome found that members of this population are not motivated to participate in research based on extrinsic rewards (Haas, et al., 2016). Instead of monetary incentives, adults with ASD in the aforementioned study wanted to be heard and “understood as equal and valued partners in research in order to both improve community understanding of ASD and to participate in the wider community” [9]. Describing the benefits of a study for participants’ peers has been described as an effective motivator for participant recruitment with hard-to-reach populations in general, through both recruitment letters as well as during ongoing conversation between researchers and participants (Bonevski, et al., 2014).
Relationships and community rapport. Developing rapport with community and organization leaders, also known as developing derived rapport, is one way to earn trust from potential participants and support recruitment efforts. These organization leaders serve as gatekeepers to their communities, and can relay information between researchers and potential participants (Ellard-Gray, et al., 2015). This can also translate to online environments and recruiting on a broader scale. In recruiting adults with ASD without intellectual disability, Haas, et al. (2016) found that effective recruitment methods included “via social media of autism support networks, support groups and service providers (with) assistance from active, high-profile and well-connected advocates in the HFA/AS community, and snowballing” [10]. Snowball sampling through social networking sites is also described as a potential avenue for successful recruitment of hard-to-reach populations, whereby researchers can post a call for participation, which can then be shared broadly across individual or institutional social media accounts. This allows potential participants to view this call for participation anonymously and contact the researcher directly (Sadler, et al., 2010).
Online forums. Adults with ASD use the Internet for connecting and engaging with their peers, finding community and an opportunity to self-advocate through online forums (Jordan, 2010). As one self-described autistic adult states: “the online autistic adult community presents an important resource for collaboration. The autistic community has a large presence on blogs, email lists, websites, and threaded online forums ...” (Robertson, 2009). One such online forum is Reddit, with has been used with some success to recruit large samples quickly and at no cost. Reddit can also be used when researchers are interested in targeting specific groups (Shatz, 2017). One study suggests using seven guidelines for recruiting from Reddit specifically: post to a relevant subreddit; post to multiple relevant subreddits when necessary; clearly highlight the nature of the study in the post’s title; post at an optimal time of day; note for demographics by time of time posting; keep surveys concise to avoid attrition; and engage with the community after posting if there are any questions (Shatz, 2017). Subreddits are forums designed for engagement around a particular topic.
Similar to a subreddit, some discussion forums are developed specifically for a member group; autism specific forums include Wrong Planet, Spectrum Forums, and Autism Forums. Some studies have used online discussion forums as an avenue for collecting data from individuals on the autism spectrum (Anderson, 2018, 2016; Jordan and Caldwell-Harris, 2012), though not always specifically for recruitment purposes.
Mechanical Turk. Amazon’s Mechanical Turk (MTurk) has increasingly been used by researchers to generate study participation, and has promise in being used to recruit hard-to-reach populations. MTurk has been described as a “one-stop shop for getting work done, bringing together the people and tools that enable task creation, labor recruitment, compensation, and data collection.” [11] Through the site, “requesters” post the task they need completed from a computer; “workers” can browse through all tasks posted and accept and complete those that are applicable to them. Upon completion, requesters review the work and provide payment, often a token amount. Requesters can deny payment for subpar work; as “requesters can limit their tasks to workers with low refusal rates,” workers have an incentive to deliver quality work [12]. MTurk has been used successfully for social science research participation, and “the quality of data provided by MTurk (meets) or exceed(s) the psychometric standards associated with published research.” [13] One major benefit of using MTurk for participant recruitment is in time savings; MTurk is far less time consuming than recruiting participants for traditional, in-person lab research, and also can be faster than social media recruitment, which has been shown as “far less immediate” than generating responses through MTurk [14].
Summary
In alignment with work in critical disability studies, librarians who wish to improve accessibility should not do so without participation and input from members of the autism community themselves. Thus, recruiting college students with autism to participate in future work is critical.
Sampling online might provide a buffer for students with ASD in that they can participate anonymously from a location of their choosing, and do not have to self-identify as “other” in front of their peers. Additionally, online recruitment efforts through discussion forums and MTurk have led to some successes for members of hard-to-reach populations.
As part of a larger project that sought to involve college students on the spectrum in the design of training for academic librarians, this segment of the study focuses only on the recruitment and survey completion of such students. An eight-member advisory board, of which four members described themselves as autistic or on the autism spectrum, met online twice while this process was underway, and was provided with updates and the opportunity to give feedback about recruitment efforts.
Survey design
A survey was developed using the Library Anxiety Scale (LAS), a measure first introduced in 1992 and used to understand how undergraduate college students feel about using their libraries (Bostick, 1992). While the LAS has since been updated to reflect the digital era (Van Kampen, 2004), the original instrument is still used to gather reliable data from college students about how they use and experience their campus libraries. In this study, the original LAS was selected for use as it was determined to be a more reliable data collection instrument. 45 total questions were included, all ranked using a 1–5 Likert scale (1 = strongly disagree; 5 = strongly agree). Permission to use the survey was given by its creator, and the survey was imported into Qualtrics survey software. Additional demographic questions were included at the beginning of the survey, as well as an open-ended question about where the participant first learned about the survey and a question if they would like to also participate in an interview, either via video, phone, or in person. A stop question was also included that asked the participant if he or she identified as a student with autism. Answering “no” to this question ended the survey.
Population, sample, and participants
The population for this study was college students from the United States who identify as being on the autism spectrum. Participants were recruited from two different frames through convenience sampling: (1) online discussion forums from Reddit and Autism Forums; and, (2) MTurk. With both sampling frames, a call for participation was posted along with a direct link to the Qualtrics survey.
First, the researchers sampled from a national audience using the online discussion forum Reddit and the online forum Autism Forums. More specifically, calls for participation were posted on the following: https://www.reddit.com/r/autism/; https://www.reddit.com/r/aspergers/; https://www.reddit.com/r/autistic/; https://www.reddit.com/r/samplesize/; https://www.reddit.com/r/neurodiversity/; and, https://www.autismforums.com/forums/. These forums were purposefully selected as they appeared to have active participation from self-advocates, as opposed to forums for parents or caregivers.
No monetary incentive was provided, though participants were informed that their contributions would lead to more informed and potentially improved practices for members of their population with language built into the consent form.
Data was collected from online discussion forums over the course of four months, though the bulk of survey responses occurred immediately after posting the call for participation.
Next, the researchers sampled using MTurk and providing a token incentive of US$.50 per survey completion. Members of the MTurk community are active due in large part to the compensation they are provided. Though often token amounts, completing a large amount of surveys can add up for diligent users. Because of this, for those who participated in the survey through MTurk, compensation was offered. Still, the same language was used for the consent form as described above and the call to action was clear in that their participation will help to inform librarians about improving services to the ASD population. Data was collected from MTurk for one day, as responses came rapidly after the call for participation and the target number of 100 was quickly reached.
Data analysis
Data was imported into SPSS from Qualtrics for analysis. To compare the mean LAS scores for participants from MTurk and from online discussion forums and in an effort to test the probability that both set of scores came from the same population, an independent-samples t-test was conducted. The significance threshold was set at .05 and equal variance assumed. To explore responses in more depth, frequencies for each survey answer were also compared and analyzed.
For the purposes of this study, success was measured by time, participant count, participant interest in following up with a second phase of the study, and representativeness of the target population. Time is determined by the amount of time that passed between calls for participation and adequate participant numbers reached, and participant count is measured by the number of survey participants that completed the survey in entirety (to note, most questions were not required by design, so that if a student with ASD felt uncomfortable at any point, he or she could skip a question. As such, total n may vary by question). Interest in following up is determined by how many participants from each sampling frame were interested in the study enough to provide contact information so that they could give even more information via interview. Representativeness of the target population is determined by comparing results to what is known about members of the population and was examined through data comparison of sampling frames.
Success of online discussion forum recruitment
Calls for participation were distributed among online discussion forums on Reddit (through multiple forum topics) and Autism Forums (through one forum topic), and were initially posted in January 2017. Continued posts calling users to action were initiated for the next four months, until May 2017.
After calls for participation, 36 individuals began the survey. Researchers individually reviewed each survey attempt to determine validity and survey completion and removed those that did not meet the study’s criteria. From this, only participants who did not complete more than half of the survey were removed. All other participants gave valid answers within open-ended demographic questions, and all claimed to be college students on the autism spectrum. As an example, “Where did you first hear about this survey” yielded expected answers such as “Reddit” and “Autism Forums,” and open-ended responses to gender yielded answers such as “nonbinary,” and “male.” Sixteen participants from online discussion forums remained and were included in data analysis.
Six of the 16 final participants, or 37.5 percent of respondents, from these surveys provided e-mail addresses to be contacted for interviews in the second phase of this study. These e-mail addresses were not utilized, so it cannot be said for certain that all e-mail accounts that were given were actually valid.
Success of MTurk recruitment
From the call for participation through MTurk, 168 participants began the survey. Of those, 100 finished the survey in entirety and were approved for payment. The call for participation on MTurk remained available for two days before the goal of 100 approved participants was reached.
Responses were reviewed critically, and through post-hoc exclusion participants were removed if they gave information in the brief, open-ended, initial demographic fields that indicated the survey was completed by a robot or by a survey taker who did not meet the criteria, did not answer truthfully or thoughtfully, and simply wanted the small incentive. An example of this is for the open-ended field that asked “how did you hear about this study,” to which some potential participants entered “good,” or a series of numbers.
A qualifying criterion at the onset of the study asked if participants were college students on the autism spectrum, and ended the survey with a thank you message if the participant chose “no.” Some participants tried to work around this and clicked “yes” to continue with the survey, and then selected “no” on a follow-up question “do you identify as being on the autism spectrum.” These surveys were not included for analysis. After careful review for validity, 78 participants from MTurk remained and were included in data analysis.
Of the 78 total participants included for analysis from the MTurk sampling frame, 22, or 28.2 percent, said that they would be interested in a follow-up interview and listed an e-mail address. Again, however, it must be noted that these addresses were not checked for validity.
Data gathered from each recruiting method
Survey data was gathered using two online methods: recruiting through online discussion forums with self-identified individuals with ASD, and recruiting through an online task/payment system, MTurk. Both recruitment attempts were with the same targeted population, college students with ASD in the United States. Both online recruitment efforts were made within a potentially international audience, though requesting that only those who met the selection criteria participate.
Comparison of means
From the independent-samples t-test, with a significance threshold of .05 and equal variances assumed, eight variables returned significant differences in the means: 1) I’m embarrassed that I don’t know how to use the library; 2) The library offers programs, services, or resources that address my needs; 3) I can’t get help in the library at the times I need it; 4) There is too much crime in the library; 5) I don’t feel physically safe in the library; 6) I get confused trying to find my way around the library; 7) The librarians are unapproachable; and, 8) The computer printers are often out of paper (Table 1).
Table 1: Mean LAS variables with significant differences. Note: Larger version of Table 1 available here.
There was no significant difference in scores for each participant group in the remaining 37 of the 45 total questions.
Comparison of frequencies
To give further insight into the eight variables in which there were statistically significant differences, frequencies were run for both sampling frames (Table 2). Data is presented using valid percentages, using the number of responses to each question and not the sample’s total, as not every participant chose to answer every question.
Table 2: Frequencies and valid percent. Note: Larger version of Table 2 available here.
Embarrassment of lack of knowledge. The most significant difference developed from the question “I am embarrassed that I don’t know how to use the library” (p < .000). Students recruited from discussion forums know how to use the library — or at least they aren’t embarrassed about their library skills, with 62.5 reporting that they “strongly disagree” with the statement “I’m embarrassed that I don’t know how to use the library,” and another 18.8 percent who “disagree” with this statement. In comparison, only 6.4 percent of participants from MTurk “strongly disagree” with the statement “I’m embarrassed that I don’t know how to use the library,” while 38.5 percent “disagree.”
Adequate services and programs. Responses were significantly different to the statement “The library offers programs, services, or resources that address my needs” (p < .006).
Students from discussion forums are not sure how to view the library in terms of programs and services. 50 percent selected “neither agree, nor disagree.” None selected “strongly agree,” and 35.7 percent chose either “disagree” or “strongly disagree.” Students from MTurk felt more positively; 50 percent selected “agree,” followed by 24.4 percent who chose “neither agree, nor disagree.”
Crime and safety. Participants differed significantly in their responses about crime and safety in the library; responses differed to both the question “I don’t feel physically safe in the library” (p < .013) and “There is too much crime in the library” (p < .013). Participants from online discussion forums responded more strongly that they do feel safe, and that there is not too much crime in the library; students from MTurk did not respond similarly.
In taking a closer look at the data, not one student from discussion forums selected either “agree” or “strongly agree” with the statement “I don’t feel physically safe in the library;” this question generated the second lowest mean (with strongly disagree representing 1 and strongly agree representing 5) from this sample in the entire study, at 1.64 percent. MTurk students largely “strongly disagree” (24.4 percent) or “disagree” (32.1 percent) with “I don’t feel physically safe in the library,” though their response to this question is not as clearly polarized.
The lowest mean response from students on online discussion forums, at 1.5 percent, skewed left toward “strongly disagree,” is in response to the question “There is too much crime in the library.” In comparison, participants from MTurk also selected “strongly disagree” (30.8 percent) and “disagree” (25.6 percent), though the responses were not as strongly skewed (x̄ = 2.42).
Approaching librarians and seeking help. The two sampling frames differed on questions related to approaching authority figures, the librarians, for help (“I can’t get help in the library when I need it”: p < .008; “The librarians are unapproachable”: p < .021).
From those recruited from online discussion forums, more than half of students think that, as a whole, librarians are unapproachable. 43.8 percent “agree” and another 25 percent “strongly agree” that “The librarians are unapproachable.” MTurk participants were spread, with 34.6 who “disagree,” 33.3 “undecided,” and 23.1 who “agree.”
However, most students from discussion forums do feel that they are able to get help when needed. Not one student chose “strongly agree” to the statement “I can’t get help in the library when I need it;” instead, 26.7 percent selected “strongly disagree” and 40 percent chose “disagree.” In contrast, students from MTurk chose both “agree” at 35.9 percent and “disagree” at 34.6 percent, and answers otherwise ran the spectrum from strongly disagree to strongly agree.
Navigating spaces. Results differed from the two sampling frames to the statement “I get confused trying to find my way around the library” (p < .020). More than half of students from discussion forums feel confident finding their way around the library. Only one survey respondent chose “strongly agree” while 26.7 percent chose “strongly disagree” and 33.3 chose “disagree.” Alternatively, more than half of the students from MTurk “agree” or “strongly agree,” at a combined 56.4 percent.
Equipment and supplies. Finally, the responses differed for the statement “The computer printers are often out of paper” (p < .047). While more than half of participants from both sampling frames “strongly disagree(d)” or “disagree(d)” (64.3 percent of participants from online discussion forums and 51.3 percent from MTurk), distribution varied greatly otherwise. Only 7.1 percent of participants from discussion forums chose “agree” and no participant selected “strongly agree.” In comparison, 26.9 percent of participants from MTurk “agree,” and 3.8 percent “strongly agree.”
Recruitment success
Researchers attempted to have 100 total survey participants from each recruiting method. While 16 total participants completed surveys from online discussion board calls for participation, this only happened after posting in multiple forums over the course of four months. In contrast, only one post was made to MTurk, and the study had to be shut down as participation numbers — and compensation allotted for participants — were quickly met. From an initial pool of 168 survey attempts, 78 were deemed to be valid participants and were included for analysis. In calculating the time invested and numbers alone, the recruiting from MTurk garnered far more participants for the amount of time and effort invested. With one post, near target participation was reached. Meeting the target sample size in a timely fashion is important for researchers’ efforts in recruitment, as barriers here can lead to a project’s delay, or sometimes even abandonment (Ashery and McAuliffe, 1992).
From sample size and timing alone, one might assume that MTurk was the superior recruitment approach. However, a slightly higher percentage of participants who responded through online discussion forums (38 percent) compared to those from MTurk (28 percent) were interested in continuing with the study as interview participants, indicating a willingness to commit greater time and energy to the research results. It should also be noted that there was no mention of additional compensation for interview participants generated through either method of online recruitment, so this was not a factor for comparison. Participant retention is also an important part of recruitment; the willingness of survey respondents to commit more time to further work demonstrates their investment into the project as a whole. Previous works indicate that adults on the autism spectrum are more likely to participate when they know that their contributions are being used to improve services for other members of the population (Haas, et al., 2016). It is possible that those who were willing to provide contact information and continue their potential for participation did so with this in mind, especially given the circumstances that no additional compensation was described.
MTurk is an avenue where people go with the explicit expectation of being crowd sourced, while online discussion forums are not designed with this as the sole purpose. People self-select into the MTurk space with the intention of contributing to online projects for a token reward. Those who choose to participate in the MTurk environment are a specific subset of the population as a whole, and cannot be reflective of members of an entire group. Instead, the data gathered from MTurk must only be generalized to the population from which it came, and is reflective only of MTurk users.
Participation and collaboration
Recruitment was more challenging, and the sample size smaller, than anticipated. The nature of this study and the manner in which it was framed might play a role into why participation was low. The autistic community has traditionally experienced more institutional oversight than neurotypicals. Traditionally, neurotypicals have spoken for, and over, individuals on the spectrum, not always representing their experiences accurately or appropriately. It is quite possible that the low response rate has something to do with this; these individuals do not want to participate in yet another initiative that seeks to represent their experiences without their full participation.
This paper developed from a larger grant funded project, which ultimately sought to design training for librarians about how to better serve their patrons on the spectrum. The project as a whole worked closely with members of an advisory board, of which multiple members identified as autistic. While their input was sought frequently, it is true that this segment of the project, that is the recruiting for and implementation of the LAS, was largely designed and implemented by the researchers, through a neurotypical lens. Instead of implementing a co-design approach, the study used a model in which the study was designed and presented for approval.
Participation rates from this study illuminates the need for better inclusion practices in future work. This study was developed by neurotypical researchers, with some input and oversight by a panel of autism self-advocates. While the input from this advisory board was extremely valuable, it was not utilized to the fullest extent. Reflecting on this study, it is clear that more participation from students on the spectrum could have been gained by framing the study with more inclusion of members of the population itself. For example, the study framed recruitment for students with “autism.” Previous work indicates that college students hold more stigmatized opinions of the term autism compared to the term Asperger’s syndrome (Butler and Gillis, 2011). As Asperger’s syndrome is no longer given as a diagnosis since the DSM-V incorporated this into autism spectrum disorder more broadly in 2013, the researchers used “correct” terminology to recruit students with “autism.” However, it is possible, and even likely, that many college students were given a diagnosis of Asperger’s syndrome and perhaps continue to identify as such. Had more students on the spectrum been involved in research design, would they have suggested using the term Asperger’s instead to draw students who identify as such? This is unknown but is an area in which more input could have improved research design.
The advisory board was asked for general approval or disproval of the instrument used, but not explicitly asked to submit their own, alternative measures. Similarly, the advisory board was presented with challenges in recruiting throughout the process and while their advice was sought, there was no explicit request of them, as members of the autistic community, to recruit students independently.
Data comparison
Sample size and retention are not enough to determine a successful participant recruitment; the sample must also provide representativeness, and accurately represent the target population (Babbie, 1998). Though this study was not conducted using a random sample and the results cannot be generalized to the population as a whole, the data can be compared to what is currently understood about members of the autism population, particularly those in college or university. Despite some similarities, some data collected from MTurk was quite different from that collected from members of the same targeted population on online discussion forums. The following compares data collected from both online methods both to one another as well as to the literature that is available regarding expectations of members of the targeted population and discusses these results.
Embarrassment and knowledge of the library. Feelings accounted for some of the significant differences between sampling frames, with one question that leads with “I feel embarrassed ...” and another that leads with “I get confused ...”. In both instances, participants from MTurk were much more like to agree, while participants from online discussion forums were much more likely to disagree; both questions need to be further examined. Are students from online discussion forums saying that they actually do know how to use the library, or just that they do not feel embarrassed by it? Similarly, are students from online discussion forums saying that they know how to find their way around the library, or that they just do not feel confused while navigating the space?
The largest difference between sampling frames is whether or not one is “embarrassed that (they) don’t know how to use the library” (p < .000); 62.5 percent of discussion forums participants strongly disagree, compared to only 6.4 percent of those from MTurk. If examining the strong responses from online discussion forums, one might question one of two things: Do students with ASD consider themselves to be strong library users? One study of how students with ASD use the library describes that library knowledge is a source of pride for some (Anderson, 2018), and other sources support the idea that students with ASD likely are comfortable using the library (Everhart and Escobar, 2018). However, it is also possible that it is the term “embarrassed” that generates such a strong response. Studies about embarrassment and autism have suggested that individuals on the spectrum might be less likely to feel embarrassed, as embarrassment requires monitoring of opinions and attitudes of others (Capps, et al., 1992). Though a more recent study suggests that individuals with ASD do feel embarrassment for others in an awkward group situation, the authors suggest that this might be due to them “pick(ing) up on cues in the scenarios presented that they have learned are associated with the emption of embarrassment and subsequently respond(ing) appropriately ... perhaps adopting a ‘primitive’ level of embarrassment.” [15]. Whether the participants from the online discussion forums were responding strongly from one stance or the other, there is support from the literature that students with ASD might indeed answer in this way to this question. Why this same strong view is not reflected in the responses gathered from MTurk is unclear.
Confusion and library spaces. More than half of students from discussion forums do not “get confused trying to find my way around the library,” while more than half of students from MTurk do. Are students from online discussion forums claiming that they are adept at navigating the space? Or is it instead a question about the feelings associated with confusion, and that they do not feel confused while they attempt to make sense of their surroundings? There is support for both concepts.
A study of wayfinding in the library with one student on the spectrum suggests that students with ASD might be confident in their ability to navigate the library (Everhart and Escobar, 2018) supporting student responses from the online discussion forums. Similar to the previous discussion about embarrassment, it might be possible that instead of responses being about navigating the library, the students are responding that they simply do not feel confused. It could be that they are not adept at wayfinding, but just do not feel confused while they make their way around the physical space. As one author notes, in individuals with ASD “who report difficulties identifying their own emotions, (the) association between subjective and objective measure of emotional responses is reduced or absent ... . They may report feeling calm when, in fact, their body is in a state of high alert.” [16] Again, responses to this question need to be examined in more depth to determine whether participants are replying about their wayfinding skills or about feelings of confusion.
Programs that “address my needs”. Responses from both sampling frames differed greatly when addressing the statement: “The library offers programs, services, or resources that address my needs.” Half (50 percent) of participants from online discussion forums were undecided, while half (50 percent) of participants from MTurk agreed.
Again, this question might need to be clarified to provide more valid results. Individuals on the autism spectrum are literal thinkers, and one of these questions asks about “needs,” but does not spell out what those needs might be. A student answering this question might be confused as to whether he or she should be responding as a person with ASD, as a college student in general, or for any other number of personal or academic reasons. Participants from this population, from either sampling frame, might not know how to interpret this question, leading to varied responses.
If in fact they are responding to services for them as students with autism, it is understandable that no one from online discussion forums would “strongly agree.” While programs for college students with autism are becoming more common, with few exceptions the campus library does not often play a role in offering targeted supports.
Approachability of authority figures and asking for help. Students from online discussion forums think that librarians are unapproachable but that they can get help in the library when needed. This complete finding is reflected in a recent study by Everhart and Escobar: “Initially, Elliot also took time to extensively search…yet once he found the student worker to be helpful and trustworthy, he returned for their assistance on subsequent tasks without hesitation” (Everhart and Escobar, 2018). Though the student in this study did not initially approach library staff, he did approach for help when needed, and found the staff member to be trustworthy. Students with ASD have also described social barriers in the academic library in another study, in which librarians are called upon to address this by “become more approachable, so students know they can go to them if they have a concern.” [17] In contrast, students from MTurk don’t feel strongly one way or the other about either topic.
Crime and safety. While students from online discussion forums felt strongly that they feel physically safe and that there is not too much crime in the library, students recruited from MTurk were less clear. Responses from online discussion forums are consistent with some experiences of students on the spectrum, who view the library as a safe place and often as a place of refuge away from overwhelming campus life (Anderson, 2018).
Limitations
While this study does lead to interesting implications for recruiting hard-to-reach populations online, there are some limitations that must be noted. First, the sample size from both leads to a limitation. While from MTurk n = 78, the discussion forum n = 16. In both cases, a larger number would be ideal, and this is particularly true for those recruited from online discussion forums. As such, generalizations cannot be made from the data, and instead are representative of the participants themselves.
Additionally, data from all discussion forums is presented in aggregate. It is possible that users of different Reddit sub-forums might contribute different data; a frequent contributor or subscriber to the /autism forum might be different then the /aspergers forum. Additionally, a member of Autism Forums online community might present a different type of user than Reddit all together.
Participant recruitment can be a challenge for researchers working to gather feedback from hard-to-reach populations, including individuals on the autism spectrum. In this study, attempts were made to recruit college students with ASD to better understand how they used their campus libraries. Recruiting online appeared to be the best option to successfully solicit survey participants, and researchers used both Amazon’s Mechanical Turk (MTurk) and posts on online discussion forums. The clearest indicators of success came from those related to quantitative data: time, and participation numbers. In this frame of reference, the most successful recruiting method online was MTurk.
Less clear is the success in representation. In 82.2 percent of the cases, there was no significant different between populations sampled from one online environment or the other. Only 17.8 percent of variables surveyed for showed statistically significant differences based on sampling frame. What could those differences signify? It is possible that both sampling frames accurately represent the target population, as characteristics of individuals with ASD do vary person to person, and the users bases of each online environment might simply have drawn a different segment of the ASD student population.
This study was neurotypically-driven in design and implementation, with some guidance from members of an advisory board on the spectrum. That is to say, no member of the research team identifies on the spectrum themselves and, to their knowledge, the survey instrument was developed by another researcher who does not identify on the autism spectrum. Furthermore, the participation rates in online discussion forums do not allow for generalizations to be made. However, while this study cannot definitively make generalizations about college students on the autism spectrum and their library anxieties, it does contribute to the general body of work about online recruiting strategies. Future work should more proactively engage autism self-advocates in every step of the research design, ensuring that not only is participation met, but also that the results are meaningful and relevant to the population itself.
About the authors
Amelia Anderson is an Assistant Professor of Library Science, in the Darden College of Education and Professional Studies at Old Dominion University.
Direct comments to: amanders [at] odu [dot] eduNancy Everhart is Professor of Library Science at the School of Information at Florida State University.
E-mail: everhart [at] fsu [dot] eduJuliann Woods is the Associate Dean for Research in the College of Communication and Information at Florida State University.
E-mail: jwoods [at] fsu [dot] edu
Acknowledgements
This work was supported by the Institute of Museum and Library Services Laura Bush 21st Century Librarian Program [RE-31-15-0129016, 2016].
Notes
1. Sydor, 2013, p. 35.
2. Kumbier and Starkey, 2016, p. 483.
3. Ibid.
4. Lawrence, 2013, p. 105.
5. Hurtado, 2012, p. x.
6. Ibid.
7. Sadler, et al., 2010, p. 369.
8. Haas, et al., 2016, p. 1,799.
9. Haas, et al., 2016, p. 1,803.
10. Haas, et al., 2016, p. 1,801.
11. Burmester, et al., 2011, p. 3.
12. Ibid.
13. Buhrmester, et al., 2011, p. 5.
14. Casler, et al., 2013, p. 2,159.
15. Hillier and Allinson, 2003, p. 590.
16. Gaigg, 2014, paragraph 5.
17. Anderson, 2016, p. 89.
References
Amelia Anderson, 2018. “Autism and the academic library: A study of online communication,” College & Research Libraries, volume 79, number 5, pp. 645–658.
doi: https://doi.org/10.5860/crl.79.5.645, accessed 10 August 2019.Amelia Anderson, 2016. “Wrong planet, right library: College students with autism spectrum disorder and the academic library,” Ph.D. dissertation, Florida State University.
American Library Association (ALA), 2019. “Core values of librarianship,” at http://www.ala.org/advocacy/intfreedom/corevalues, accessed 21 July 2019.
Rebecca Ashery and William McAuliffe, 1992. “Implementation numbers and techniques in randomized trials of outpatient psychosocial treatments for drug abusers: Recruitment of subjects,” American Journal of Drug and Alcohol Abuse, volume 18, number 3, pp. 305–329.
doi: https://doi.org/10.3109/00952999209026069, accessed 10 August 2019.Autistic Self Advocacy Network (ASAN), 2019. “About,” at https://autisticadvocacy.org/about-asan/, accessed 21 July 2019.
Earl Babbie, 1998. The practice of social research. Eighth edition. Belmont, Calif.: Wadsworth.
Billie Bonevski, Madeleine Randell, Chris Paul, Kathy Chapman, Laura Twyman, Jamie Bryant, Irena Brozek, and Clare Hughes, 2014. “Reaching the hard-to-reach: A systematic review of strategies for improving health and medical research with socially disadvantaged groups,” BMC Medical Research Methodology, volume 14, number 1, at https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-14-42, accessed 10 August 2019.
doi: https://doi.org/10.1186/1471-2288-14-42, accessed 10 August 2019.Sharon Bostick, 1992. “The development and validation of the Library Anxiety Scale,” Ph.D. dissertation, Wayne State University).
Michael Buhrmester, Tracy Kwang, and Samuel Gosling, 2011. “Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data?” Perspectives on Psychological Science, volume 6, number 1, pp. 3–5.
doi: https://doi.org/10.1177/1745691610393980, accessed 10 August 2019.Robert Butler and Jennifer Gillis, 2011. “The impact of labels and behaviors on the stigmatization of adults with Asperger’s disorder,” Journal of Autism and Developmental Disorders, volume 41, number 6, pp. 741–749.
doi: https://doi.org/10.1007/s10803-010-1093-9, accessed 10 August 2019.Lisa Capps, Nurit Yirmiya, and Marian Sigman, 1992. “Understanding of simple and complex emotions in non-retarded children with autism,” Journal of Child Psychology and Psychiatry, volume 33, number 7, pp. 1,069–1,182.
doi: http://dx.doi.org/10.1111/j.1469-7610.1992.tb00936.x, accessed 10 August 2019.Krista Casler, Lydia Bickel, and Elizabeth Hackett, 2013. “Separate but equal? A comparison of participants and data gathered via Amazon’s MTurk, social media, and face-to-face behavioral testing,” Computers in Human Behavior, volume 29, number 6, pp. 2,156–2,160.
doi: https://doi.org/10.1016/j.chb.2013.05.009, accessed 10 August 2019.Joyce Davidson and Victoria Henderson, 2010. “‘Coming out’ on the spectrum: Autism, identity and disclosure,” Social & Cultural Geography, volume 11, number 2, pp. 155–170.
doi: https://doi.org/10.1080/14649360903525240, accessed 10 August 2019.Amy Ellard-Gray, Nicole Jeffrey, Melissa Choubak, and Sara Crann, 2015. “Finding the hidden participant: Solutions for recruiting hidden, hard-to-reach, and vulnerable populations,” International Journal of Qualitative Methods, volume 14, number 5 (17 December).
doi: https://doi.org/10.1177/1609406915621420, accessed 10 August 2019.Nancy Everhart and Kristie Escobar, 2018. “Conceptualizing the information seeking of college students on the autism spectrum through participant viewpoint ethnography,” Library & Information Science Research, volume 40, numbers 3–4, pp. 269–276.
doi: https://doi.org/10.1016/j.lisr.2018.09.009, accessed 10 August 2019.Jean Faugier and Mary Sargeant, 1997. “Sampling hard to reach populations,” Journal of Advanced Nursing, volume 26, number 4, pp. 790–797.
doi: https://doi.org/10.1046/j.1365-2648.1997.00371.x, accessed 10 August 2019.Sebastian Gaigg, 2014. “People with autism don’t lack emotions but often have difficulty identifying them,” The Conversation (7 April), at http://theconversation.com/people-with-autism-dont-lack-emotions-but-often-have-difficulty-identifying-them-25225, accessed 1 April 2018.
David Giles, 2014. “‘DSM-V is taking away our identity’: The reaction of the online community to the proposed changes in the diagnosis of Asperger’s disorder,” Health, volume 18, number 2, pp. 179–195.
doi: https://doi.org/10.1177/1363459313488006, accessed 10 August 2019.Karen Haas, Debra Costley, Marita Falkmer, Amanda Richdale, Kate Sofronoff, and Torbjörn Falkmer, 2016. “Factors influencing the research participation of adults with autism spectrum disorders,” Journal of Autism and Developmental Disorders, volume 46, number 5, pp. 1,793–1,805.
doi: https://doi.org/10.1007/s10803-016-2708-6, accessed 10 August 2019.Douglas Heckathorn, 1997. “Respondent-driven sampling: A new approach to the study of hidden populations,” Social Problems, volume 44, number 2, pp. 174–199.
doi: https://doi.org/10.2307/3096941, accessed 10 August 2019.Ashleigh Hillier and Lesley Allinson, 2003. “Understanding embarrassment among those with autism: Breaking down the complex emption of embarrassment among those with autism,” Journal of Autism and Developmental Disorders, volume 32, number 6, pp. 583–592.
doi: https://doi.org/10.1023/A:1021259115185, accessed 10 August 2019.Sylvia Hurtado, 2012. “Forward,” In: Terrell Strayhorn (editor). College students’ sense of belonging: A key to educational success for all students. New York: Routledge, pp. ix–x.
Jaci Huws and Robert Jones, 2008. “Diagnosis, disclosure, and having autism: An interpretative phenomenological analysis of the perceptions of young people with autism,” Journal of Intellectual and Developmental Disability, volume 33, number 2, pp. 99–107.
doi: https://doi.org/10.1080/13668250802010394, accessed 10 August 2019.Chloe Jordan, 2010. “Evolution of autism support and understanding via the World Wide Web,” Intellectual and Developmental Disabilities, volume 48, number 3, pp. 220–227.
doi: https://doi.org/10.1352/1934-9556-48.3.220, accessed 10 August 2019.Chloe Jordan and Catherine Caldwell-Harris, 2012. “Understanding differences in neurotypical and autism spectrum special interests through Internet forums,” Intellectual and Developmental Disabilities, volume 50, number 5, pp. 391–402.
doi: https://doi.org/10.1352/1934-9556-50.5.391, accessed 10 August 2019.Alana Kumbier and Julia Starkey, 2016. “Access is not problem solving: Disability justice and libraries,” Library Trends, volume 64, number 3, pp. 468–491.
doi: https://doi.org/10.1353/lib.2016.0004, accessed 10 August 2019.Emily Lawrence, 2013. “Loud hands in the library: Neurodiversity in LIS theory & practice,” Progressive Librarian, volume 41, pp 98–109, and at http://www.progressivelibrariansguild.org/PL/PL41/098.pdf, accessed 10 August 2019.
Scott Michael Robertson, 2009. “Neurodiversity, quality of life, and autistic adults: Shifting research and professional focuses onto real-life challenges,” Disability Studies Quarterly, volume 30, number 1, at http://dsq-sds.org/article/view/1069, accessed 10 August 2019.
doi: http://dx.doi.org/10.18061/dsq.v30i1.1069, accessed 10 August 2019.Georgia Robins Sadler, Hau-Chen Lee, Rod Seung-Hwan Lim, and Judith Fullerton, 2010. “Recruitment of hard‐to‐reach population subgroups via adaptations of the snowball sampling strategy,” Nursing & Health Sciences, volume 12, number 3, pp. 369–374.
doi: http://dx.doi.org/10.1111/j.1442-2018.2010.00541.x, accessed 10 August 2019.Itamar Shatz, 2017. “Fast, free, and targeted: Reddit as a source for recruiting participants online,” Social Science Computer Review, volume 35, number 4, pp. 537–549.
doi: https://doi.org/10.1177/0894439316650163, accessed 10 August 2019.Anna Sydor, 2013. “Conducting research into hidden or hard-to-reach populations,” Nurse Researcher, volume 20, number 3, pp. 33–37.
doi: https://doi.org/10.7748/nr2013.01.20.3.33.c9495, accessed 10 August 2019.Doris Van Kampen, 2004. “Development and validation of the multidimensional library anxiety scale,” College & Research Libraries, volume 65, number 1, pp. 28–34.
doi: https://doi.org/10.5860/crl.65.1.28, accessed 10 August 2019.Susan W. White, Thomas H. Ollendick, and Bethany C. Bray, 2011. “College students on the autism spectrum: Prevalence and associated problems,” Autism, volume 15, number 6, pp. 683–701.
doi: https://doi.org/10.1177/1362361310393363, accessed 10 August 2019.
Editorial history
Received 7 May 2019; revised 25 July 2019; accepted 8 August 2019.
This paper is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.Autism and online recruiting methods: A comparison of Mechanical Turk and discussion forums
by Amelia Anderson, Nancy Everhart, and Juliann Woods.
First Monday, Volume 24, Number 9 - 2 September 2019
https://firstmonday.org/ojs/index.php/fm/article/download/10105/8068
doi: http://dx.doi.org/10.5210/fm.v24i9.10105
