Although many contemporary students feel comfortable searching the web and finding information,¹ they may not routinely analyze the accuracy of what they have found—at least in my experience. Of course, this perspective is far from new; for example, twenty years ago, Graham and Metaxas discussed the student perspective that, "of course it's true, I saw it on the internet" [3], and various approaches have been discussed to address this issue (e.g., see [7,9]).

Overall, faculty have been trying to teach this skill of analyzing sources for many years. Recently, the introduction of ChatGPT and other Large Language Models (LLMs) seems to have returned this topic to the forefront of educational discussion (e.g., [4,6]).

To address the challenge of helping students find and analyze sources—a topic sometimes included within discussions of information literacy,² this column is organized into three main sections:

1. an historical perspective on students' practices and approaches for finding and analyzing information;
2. the need for generating interrupts to help students confront common practices and attitudes in obtaining information; and
3. a description of an updated approach involving a series of research exercises, building on experiences described in [9], integrating additional perspectives from a range of sources, including [5,8].

Some Historical Context

Back in 2003, Graham and Metaxas conducted a survey of student practices and perceptions regarding finding answers to a variety of questions. Several of their main conclusions follow.

---

Faculty have been trying to teach this skill of analyzing sources for many years. Recently, the introduction of ChatGPT and other Large Language Models (LLMs) seems to have returned this topic to the forefront of educational discussion …

---

• Students utilized the internet almost exclusively ("less than 2% of students' responses to all questions included non-Internet sources") [3, p. 72].
• "Students were overwhelmingly susceptible to three types of misinformation—advertising claims, government misinformation, and propaganda—and somewhat susceptible to scam sites" [3, p. 73].
• Most students depended upon the first answer they found in an internet search without double checking (75% on one survey question and 78% on a second question) [3, p. 74].
• "Students in this study seemed to have a great deal of confidence in their abilities to distinguish the good sites from the bad" [3, p. 75].

Since that time, my own experiences with students have been quite consistent with these findings, although some interactions suggest that many of today's students may be even more dependent upon the internet, less aware of what sites might be trusted for information, and largely unaware of the need to analyze and evaluate information sources. Three examples (from my experiences at different schools) illustrate understandings and practices that seem common.

• In researching current capabilities of artificial intelligence for a paper in a non-majors course, a student cited the movie, 2001 A Space Odyssey, to identify technical accomplishments in the field.
• In an upper-level course on Algorithmic Analysis, a student repeatedly challenged the basic definitions of Big-Ω, Big-Θ, and Big-O and associated examples, because in-class statements contradicted claims from several internet blogs; the student believed the blogs over the textbook and the instructor, saying "why would they [the bloggers] lie?"
• In a lab-based CS1 course, students worked through exercises, that drew on readings online. Several students commented on the difficulty of the exercises because they could not find answers through web searches. When directed to the online course readings, they indicated it had not occurred to them to check the course materials—but students noted the exercises were much easier once they utilized course-related readings.

Although these anecdotes are far from a careful statistical study, many such examples suggest that current students are no more sophisticated in their searches for material than those 20 years ago in the Graham-Metaxas study. One might speculate that experiences with remote learning during the Covid-19 epidemic may have compounded the problems, but such research is beyond the scope of this column.

In addition, the emergence of ChatGPT and similar packages may provide new mechanisms for students searching for information. Although various tools have different interfaces and internal algorithms, underlying issues of locating information and analyzing answers remain as in the past. In recent months, numerous articles and communications have discussed both the capabilities and shortcomings of these online systems. Two quotes highlight some of the on-going challenges.

• The first by Gary Marcus in the Communications of the ACM places an example within a broader context.
  These systems are inherently unreliable, frequently making errors of both reasoning and fact, and prone to hallucination (http://bit.ly/3wL4lr4); ask them to explain why crushed porcelain is good in breast milk (https://bit.ly/3Riw8xv), and they may tell you that "porcelain can help to balance the nutritional content of the milk, providing the infant with the nutrients they need to help grow and develop." [6, p. 6]
• The following interaction with ChatGPT, reported by Brian Blitz in MAA Connect [2], highlights errors in logical reasoning:

Does 6 divide n(n^2+5)?

CHATGPT

To determine if 6 divides N(N^2+5), we need to consider the possible remainders when N is divided by 6.

When N is divisible by 6, then N(N^2+5) is also divisible by 6, since both terms contain a factor of 6.

When N leaves a remainder of 1 or 5 when divided by 6, we have:

N ≡ 1 (mod 6) → N^2 ≡ 1 (mod 6) → N^2+5 ≡ 0 (mod 6) N ≡ 5 (mod 6) → N^2 ≡ 1 (mod 6) → N^2+5 ≡ 0 (mod 6) In either case, N(N^2+5) is divisible by 6.

When N leaves a remainder of 2 or 4 when divided by 6, we have:

N ≡ 2 (mod 6) → N^2 ≡ 4 (mod 6) → N^2+5 ≡ 3 (mod 6) N ≡ 4 (mod 6) → N^2 ≡ 4 (mod 6) → N^2+5 ≡ 3 (mod 6) In either case, N(N^2+5) is not divisible by 6.

Finally, when N leaves a remainder of 3 when divided by 6, we have:

N ≡ 3 (mod 6) → N^2 ≡ 3 (mod 6) → N^2+5 ≡ 2 (mod 6) In this case, N(N^2+5) is not divisible by 6.

Therefore, we can conclude that 6 divides N(N^2+5) if and only if N is divisible by 6 or leaves a remainder of 1 or 5 when divided by 6.

Of course, the error here is the assumption that integer a divides product bc only if a divides b or a divides c. This is true for primes, but not for non-prime integers, and Blitz's original query is true for all positive integers n.

The Need for Generating Interrupts to Help Students Confront Common Practices and Attitudes in Obtaining Information

Overall, both the Graham-Metaxas study [3] and anecdotal examples indicate that students utilize web searches as a primary way of obtaining information, but generally students are not very good at analyzing the material they get. They often do not consider possible biases, underlying agendas, or even basic correctness in materials obtained, and they frequently do not seek multiple sources to verify information.

---

Several students commented on the difficulty of the exercises because they could not find answers through web searches. When directed to the online course readings, they indicated it had not occurred to them to check the course materials—but students noted the exercises were much easier once they utilized course-related readings.

---

However, many students seem quite confident in their abilities to find information over the web—they often believe their current practices work well.

Historically, one approach to improve student skills has involved inviting librarians or other experts to a class. In-class presentations may outline search strategies, discuss points of view, review mechanisms to verify information, etc. For example, at Grinnell College, first-year tutorials throughout the school included discussion on research techniques—often with librarian-led class discussions, worksheet exercises, or other activities. In my experience, students listen politely to the presentations, discuss, and complete worksheets, but these activities had little, if any, long-term impact.

Altogether, students believed they knew how to find reliable answers effectively and efficiently. Presentations and in-class activities might provide a few ideas, and students could get started quickly if a talk helped them with an assignment on a new topic. However, for most work, students seemed comfortable relying on the web searches they had utilized for years.

Within this context, developing student skills to locate legitimate information likely requires at least two parts:

• demonstrating that current student approaches for [web] searching can be inadequate—sometimes retrieving incorrect, incomplete, or misleading material, and
• then providing practice in improved approaches for researching a topic.

The important point here is that the first step for improving information literacy likely involves generating interrupts—current processing may yield unacceptable results, making students receptive to ideas for information analysis and review.

An Updated Approach involving Research Exercises

To generate interrupts involving web materials, a basic approach presents students with a multi-week series of [short] homework assignments, in-class reporting, and class discussions. Instructions for the first assignment³ include the following text.

Course work in college often requires students to investigate topics using sources outside of the classroom. Since incoming students often have differing experiences in conducting such research, this exercise asks you to answer two questions using whatever techniques you are familiar with from past work. In addition, you will need to report on what process you used to find these answers.

Assuming the course meets at least twice a week (e.g., Tuesdays and Thursdays), work proceeds as follows:

Week 1:

• Tuesday: Assign Round 1 problems: Two exercises for written homework, due Thursday.
  • Each exercise requires students to locate information, provide an answer in a well-written paragraph, and include citations for each source used.
  • Students also write an additional paragraph describing the process that they followed in finding their answers. This paragraph should indicate what strategies they used in planning their search, what source(s) they consulted and why, and which source(s) were most helpful? Least helpful?
• Thursday: Round 1 answers due; In-class discussion.
  • Students present their answers (either on the board or with oral reporting).
  • Class discussion reviews answers—not focusing on individuals, but rather on the conclusions stated together with possible errors, misconceptions, omissions, etc.
  • A librarian attends, taking notes about what processes students used, together with advantages and shortcomings of the student approaches. Typically, the librarian says rather little here, but gathers insights for the next session.

Week 2:

• Tuesday: Librarian provides feedback on Round 1, and Round 2 assignments distributed.
  • The librarian reviews the strategies utilized in the first round of questions, indicates positives of some approaches, identifies shortcomings, and suggests additional strategies and resources.
  • Class discussion regarding the librarian's comments, questions, feedback, etc.
  • Distribution of two more exercises for a Round 2 written assignment
  • Thursday: Round 2 answers due; in-class discussion
  • Presentation of answers and class discussion, similar to Round 1
  • The librarian may or may not attend.

---

Students believed they knew how to find reliable answers effectively and efficiently. Presentations and in-class activities might provide a few ideas, and students could get started quickly if a talk helped them with an assignment on a new topic. However, for most work, students seemed comfortable relying on the web searches they had utilized for years.

---

Week 3:

• Tuesday: Round 3 written assignment distributed.
  • Class discussion regarding the librarian's comments, questions, feedback, etc.
  • The librarian may or may not provide further feedback and suggestions.
  • Thursday: Round 2 answers due; in-class discussion
  • Presentation of answers and class discussion, similar to Rounds 2 and 3

Although a fourth round of questions might be considered, my experience suggests that many students seem less motivated and engaged for a Round 4. Many indicated that a fourth round seemed unnecessary and largely redundant. Altogether, this approach seems to work well through three rounds of questions, but subsequent rounds may be subject to diminishing returns.

Additional notes on this approach generally fall into three categories:

• Selecting questions,
• Student reactions, and
• Future updates/adjustments (e.g., responding to social media, blogs, information silos, and ChatGPT)

---

A number of students, particularly first- and second-year students, have gained a deeper appreciation of the need to dig into questions. At first, some students may have felt ambushed by seemingly simple questions, but over the multiple problem sets they also realized that apparently straight forward questions may not have simple or easy answers.

---

• Selecting Questions

A key to this approach to promote information literacy involves the selection of questions. Over the years, several guidelines have been helpful.⁴

• A question should be reasonably simple to state and understand—at least on the surface, although some subtleties may arise upon initial investigation.
• A question might be stated, so that a naive web search yields contrasting or inconsistent answers.
• The mix of questions should involve a mix of topics and require consultation into different sources. For example, in my past research sets, one question typically involved computing in some way, while the second was non-technical.
• Questions should be reasonably specific, requiring definite statements or conclusions. Vague, abstract, or high-level answers should be discouraged.
• Although topics may touch upon controversial or popular topics for debates, the questions should focus on obtaining facts rather than opinions. Questions should promote information literacy rather than encourage political or social debates.
• To discourage students in one course from using answers from students in previous courses, an instructor should consider developing new questions each semester.
• In the past, this author's experience has found librarians to be interested in collaborating in finding appropriate questions and eager to help in this venture.

The following annotated examples illustrate questions that have worked well over several course offerings.

• "Name three innovations of Microsoft."
  • In [3], Graham and Metaxas limited this question to "innovations in the past 10 years".
  • Although a seemingly straight forward question, the term, "innovation," is not defined and thus a careful answer likely requires some analysis.
  • As in [3], this author has found that many students answer this question by going directly to Microsoft.com.
  • When asked in the first set of Research Questions, students have reported that Microsoft has innovated much of computing, including graphical user interfaces, use of a mouse, operating systems, word processing packages, spreadsheets, etc.—conclusions that have led to extensive in-class discussions!
• "Is the population of Lisbon, Portugal under 1,000,000?"
  • Graham and Metaxas asked a similar question related to area rather than population in [3].
  • Although another seemingly simple question of fact, two interpretations are possible. In one class, half of the students responded "yes"—citing a population of about 500,000, while the others responded "no"—citing a population of about 3,000,000. This led to considerable class discussion, until students realized that the populations correspond to the city limits and metropolitan area, respectively.
• "What is the most widely used computer programming language?" or "What computer programming language has been most influential?"
  • Students often find either or both of these questions of considerable interest.
  • A web search for either question typically yields a rather wide range of answers, raising questions of how to approach the answers. For example, in determining use of a language, should one consider lines of code produced in the past, ads for job openings, etc.? For computer languages, does the application area matter (e.g., databases, web sites, real-time embedded systems, etc.)?

• Student Reactions

As stated earlier, one goal of these research exercises has been to generate an interrupt for students to review how they locate and analyze information. The first iterations of these approach, as discussed in [9], reported the following.

Students become much more critical of both questions and sources. A significant percentage (perhaps 30%) of beginning-level students report that the research exercises have transformed the way they think of resource material and the strategies they use in exploring new research questions. (The remaining 70% indicate the exercises were useful if not transforming.)

Student reactions in subsequent courses using this approach have been similarly positive. Students have found the questions motivating. They have been willing to devote time, although time required per question has been reasonable. Further, many report that they have changed their perspectives based on this experience.

Further, a number of students, particularly first- and second-year students, have gained a deeper appreciation of the need to dig into questions. At first, some students may have felt ambushed by seemingly simple questions, but over the multiple problem sets they also realized that apparently straight forward questions may not have simple or easy answers.

Overall, students have indicated consistently that the time spent on homework assignments has been worthwhile, and that their approaches to finding information had matured considerably.

---

At some point in the future, experience, policies, and practices may clarify the proper role(s) of artificial intelligence in yielding true and useful material for decision making. However, at present, the web contains a remarkable amount of material, some clearly based on fact and logic, and some on fancy and/or ideology. Regardless of its source, much currently available web material may be difficult to evaluate.

---

Future Adjustments (e.g., responding to social media, blogs, information silos, and ChatGPT)

Although this approach of research exercises has worked quite well over the years, adjustments likely should be contemplated as the web landscape continues to evolve. On the positive side, the approach is flexible, and new questions can help address a range of contemporary topics and challenges. Beyond those details, three new adjustments might be considered to the original plan of three problem sets.

1. A third question might be added to each problem set, focusing on the strengths and weaknesses of ChatGPT and/or similar LLMs.
   • One question might present an interaction with ChatGPT, such as given earlier about whether 6 divide n(n^2+5) [2]. Numerous respondents on MAA connect have suggested asking whether a given ChatGPT answer obtained is trustworthy.
   • Michael Kerbs has suggested, "Ask students to pose a conceptual homework problem (e.g., "Prove that…") to the chatbot, then ask students to determine whether its response is correct. This could be an efficient way to generate "evaluate-the-answer" questions. [5]
   • One question might ask a group of 2–3 students to submit a given question to ChatGPT in several different ways, using different wordings. Responses could be compared and analyzed." [5]
2. A class session might be added to the schedule to consider ways to determine when web sources are trustworthy. This might be a class discussion, a talk from an expert on digital media, a series of talks from students reporting their explorations of the subject, or other creative formats. Although the librarian's presentation (already discussed above) might include some perspectives on this subject, a complementary session might be well advised. Some topics to be addressed might include the following.
   • In the past, if one consulted 3 or 4 sites and found general agreement, students might be able to conclude the information was valid. Today, however, material on one site might be copied through social media and blogs, creating information silos. How should sites be evaluated?
   • Some sites include comments contributed from many sources. In such cases, some statements may be insightful and fact-based, but others may be misleading, partially incorrect, or blatantly false. How can the validity of statements from different contributors be analyzed?
3. An instructor might prepare a live demonstration of answering one or two research questions in class—perhaps after Round 1 or Round 2.
   • After students have used their ad hoc approaches to finding information in Round 1, and after the follow-up in-class discussion, students likely have experienced the interrupt described earlier in this article. This might provide an environment where students would be receptive to seeing other approaches in action.
   • A live demonstration would allow an instructor to seek suggestions from students, try contrasting approaches that have varying success, and highlight effective practices.

In addition to requiring students to come to class prepared to discuss the answers they have found, each assignment also has required that students submit in written form both their answers and their process for finding the answers. In practice, these written statements are reasonably short and can be reviewed in only a modest amount of time. (Also, the collaborating librarian can use the written statements in Round 1 for the following up talk in Week 2.) In this context, an instructor not only can comment upon answers and processes, but also on writing mechanics, citation formats, and reference sections. Although this type of feedback goes well beyond matters of information literacy, such comments can have considerable impact on student understandings of technical writing and academic honesty.

Conclusion

In contemporary society, information literacy (or the ability to locate, evaluate, and use information) has become an increasingly important, but also a controversial skill. News broadcasts and articles in the public press regularly discuss misinformation, and groups debate what is and is not true. Articles in professional and general magazines regularly tout the advantages and threats of ChatGPT and other Large Language Models (LLMs).

At some point in the future, experience, policies, and practices may clarify the proper role(s) of artificial intelligence in yielding true and useful material for decision making. However, at present, the web contains a remarkable amount of material, some clearly based on fact and logic, and some on fancy and/or ideology. Regardless of its source, much currently available web material may be difficult to evaluate.

Within this context, many [most?] students have come to rely upon the web for information, but they often latch on to whatever they find quickly, and they may not evaluate what they find. Altogether, students may not realize deficiencies in their approaches to searching the web.

The research exercises discussed here focus on helping students confront possible deficiencies in their current processes and highlight ways to evaluate and assess. Of course, this approach is quite general and can be applied in many contexts and disciplines. However, specific uses of this approach have been applied specifically in computing-related courses for about two decades and thus certainly apply to computing. (A somewhat broader discussion of the subject may be found in [8], checking the index for such topics as "research exercises," "homework," "assignments," or "writing.")

Altogether research exercises seem a worthwhile and flexible way to improve the information literacy of [computing] students within current courses and with only moderate instructor time and effort.

Acknowledgments

Thanks to Takis Metaxas, Wellesley College, who helped shape my thinking about this subject through his article [3] and his discussions at the Liberal Arts Computer Science Consortium. Thanks also to Barbara Chamberlain at the Western Institute of Technology at Taranaki, New Zealand, for helping me brainstorm during early development of this approach. Special thanks to Kevin Engel, Librarian Grinnell College, who collaborated with me in developing and implementing several iterations of research exercises for use with first-year students, as discussed in [9].

References

1. Association of College and Research Libraries, A Division of the American Library Association, Presidential Committee on Information Literacy: Final Report January 10, 1989, https://www.ala.org/acrl/publications/whitepapers/presidential; accessed 2023 May 29.

2. Blitz, B., Response to the prompt, "What creating ways are you using ChatGPT to be more efficient:", in MAA Connect, April 29, 2023, 4:00 pm.

3. Graham, L., and P.T. Metaxas. Of course it's true; I saw it on the internet, Communications of the ACM, 46, 5 (2003): 71–75.

4. Klepper, D., and Valdes, M., Digital Literacy: Can the Republic 'Survive an Algorithm?', The Associated Press, March 20, 2023, https://apnews.com/article/misinformation-education-election-vaccine-canada-finland-e7d6eed63f3db289bf887605c99ab5c5 accessed 2023 May 23. Reported in ACM News on March 20, 2023.

5. Krebs, M., Response to the prompt, What creating ways are you using ChatGPT to be more efficient?, in MAA Connect, May 4, 2023, 7:30 pm PDT.

6. Marcus, G., Hoping for the Best as AI Evolves, Communications of the ACM, 66,4 (2023), 6–7.

7. Walker, H. M, "Homework assignments and Internet sources," ACM Inroads, 4,4 (2013): 16–17.

8. Walker, H, M., Teaching Computing: A practitioner's Perspective, CRC Press, Taylor and Francis Group, 2018.

9. Walker, H. M., and Engel, K, Research exercises: immersion experiences to promote information literacy, Journal of Computing Sciences in Colleges, 21,4 (2006), 61–68.

Author

Henry M. Walker
Department of Computer Science
Noyce Science Center
Grinnell College
1116 Eighth Avenue
Grinnell, Iowa 50112 USA
[email protected]

Footnotes

1. Throughout, this column discusses material (concepts, themes, history, principles, techniques, etc.) that is separate from code. Currently, the author is developing a separate column to discuss code in the context of efficiency—including code found on the web.

2. According to the Association of College and Research Libraries of the American Library Association, "to be information literate, a person must be able to recognize when information is needed and have the ability to locate, evaluate, and use effectively the needed information" [1].

3. The instructions quoted here come from a course for first-semester, first-year students. Only slight editing might be needed for courses involving students at varying levels.

4. The first three guidelines given here represent a refinement and extension of those stated in [9]. The remaining four guidelines have evolved from subsequent experience.

Copyright held by author.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2023 ACM, Inc.

Contents available in PDF
View Full Citation and Bibliometrics in the ACM DL.  

To comment you must create or log in with your ACM account.