Skip to Main Content

Generative AI Literacy

Why do I need to evaluate GenerativeAI tools?

One of the ways that you can address some of the ethical concerns around is by carefully reviewing Generative AI tools, reading about their strengths and weaknesses, and selecting a tool to use that minimizes risk while maximizing the benefit of use.

Consider the ROBOT test, described below, as a framework for evaluating whether or not you should use a particular Generative AI tool.

The ROBOT Test

Watch the following video for a brief overview of the ROBOT test, a tool for evaluating Generative AI tools. Questions to ask yourself are listed below the video.

Problems viewing? Watch Can You Trust AI Output? Use the ROBOT Evaluation Framework (ADKAR – Knowledge)(3:26 min) by Karen Ferris on YouTube

The LibrAIry, a team led by librarians from McGill University, have created the ROBOT test to help evaluate the legitimacy of a particular AI technology. It stands for:

Reliability

  • How reliable is the information available about the AI technology?
  • If it’s not produced by the party responsible for the AI, what are the author’s credentials?
  • Bias?
  • If it is produced by the party responsible for the AI, how much information are they making available?
  • Is information only partially available due to trade secrets?
  • How biased is they information that they produce?

Objective

  • What is the goal or objective of the use of AI?
  • What is the goal of sharing information about it?
    • To inform?
    • To convince?
    • To find financial support?

Bias

  • What could create bias in the AI technology?
  • Are there ethical issues associated with this?
  • Are bias or ethical issues acknowledged?
    • By the source of information?
    • By the party responsible for the AI?
    • By its users?

Ownership

  • Who is the owner or developer of the AI technology?
  • Who is responsible for it?
    • Is it a private company?
    • The government?
    • A think tank or research group?
  • Who has access to it?
  • Who can use it?

Type

  • Which subtype of AI is it?
  • Is the technology theoretical or applied?
  • What kind of information system does it rely on?
  • Does it rely on human intervention?

Attribution

The ROBOT Test is adapted from "Generative AI and ChatGPT" by The University of British Columbia Library, licensed under CC BY-NC-SA 4.0. /  A derivative of  "The ROBOT Test" by S. Hervieux & A. Wheatley via The LibrAIry, which is licensed under CC BY-NC-SA 4.0. / Adapted to Georgian context. Updates to headings and remixed info to focus on evaluating AI tools.

To cite this tool in APA format: Hervieux, S. & Wheatley, A. (2020). The ROBOT test [Evaluation tool]. The LibrAIry. https://thelibrairy.wordpress.com/2020/03/11/the-robot-test