Jean O'Grady for LTH/CHECKLIST FOR DEVELOPING A RESEARCH / LIBRARY GENERATIVE AI PROCUREMENT PLAN

Checklist for Developing a Research / Library Generative AI Procurement Plan

Published on 2024-05-31 byJean O'Grady
logo

This checklist assumes that the reader has a basic understanding of key concepts related to Generative AI  (GAI) including Large Language Models (LLM), Machine Learning, neural networks, deep learning and retrieval augmented generation [Editor's note: . Many vendors and legal tech related organizations including the American Association of Law Libraries offer webinars and resources to their members or customers.

In response to discussions with colleagues and my own discussions with vendors I have outlined below the key issues to be considered in planning for the acquisition of GAI products related to legal research. Every organization is different and my hope is that this list can provide an outline which can be adapted to the specific issues and needs across law firm, government and academic law libraries.

 

Outline of Generative AI Acquisition Issues

 

Getting a “seat at the table.” Is there a different stakeholder mix for GAI? 

Due to the complex mix of technical, ethical, training and client issues, many firms have established committees to assess the introduction of Generative AI products, often including but not limited to involvement by the following stakeholder groups:

  • Library/KM,
  • IT,
  • Professional Development,
  • IT Security,
  • Counsel’s Office General,
  • Innovation,
  • Practice Group Leaders or Innovation Committee,
  • Client Intake.

[Editor's note: LTH Survey results from Jan 2024 show that in most AmLaw 100 firms the Innovation department leads the firm's generative AI taskforce or committee, with KM and IT vying for second place]

If this is a research tool, a leader on the library or research team must have a seat at the table.

Why are you looking at GAI Products? Your assessment issues will change depending on why you are starting to look at GAI products.

  • Just exploring / experimenting
  • Looking to address specific workflow issues
  • Responding to client demand for more efficiency
  • Firm has a commitment to transform how lawyers work and  is willing to get rid of the billable hour

 

 

Dealkillers – There are several issues that stand out as potential dealkillers. These should be addressed before proceeding with a vendor or product

 

  • Can the content be trusted?
  • Security Compliance
  • GC Ethical Concern’s, Client Concerns, Court Rules restricting GAI

 

Content

    • What legal content was the product trained on? Trusted sources? Is it continuously trained as the law changes?
    • What are your processes for updating the training data? How often?
    • Are there gaps in content? E.g. dates, types of content e.g. caselaw, statutes, regulations.
    • How are content gaps addressed in training?

Security

    • Can the company pass the firm’s security compliance standards? SOC2, ISO2701 certifications.
    • If deployed in the cloud – who hosts ? Amazon Web Services. Microsoft?
    • Where are the servers located?
    • Does the solution support single sign on and multi-factor authentication?
    • Can they guarantee that your data wont be used to train the LLM?
    • Can they guarantee that uploaded documents will be purged immediately after ingestion and analysis?

 GC Ethical Concern’s and Client Concerns. What Risks are unique to GAI?

    • Fear of hallucinated answers common in Open Source platforms such as GPT 3.
    • GAI is a black box – so is natural language research!
    • Ingesting documents is not safe – but Brief checkers and Deal analyzers have been ingesting documents for years.
    • Results may not be complete and reliable – Did anyone have a policy warning lawyers not to use Google Scholar where results may also be released? This is similar.
    • The limitations of GAI can be overcome with proper “research Hygiene.”

 

Company

  • How long has the company been in the market?
  • If a private company, how are they financed?
  • How many subscribers/beta testers do they have in the legal market? Any of our peer firms?
  • What type of licenses does the company offer? Enterprise only? Practice group? Limited number of licences which are priced by tiers?
  • Product trials – free or paid? What type of restrictions in terms of people or duration?
  • Can we get testimonials or talk to other subscribers?
  • Do you have a customer user group? Or Advisory board?
  • What do your colleagues say?

Use Cases

  • What use cases does the product support?
  • Common Use Cases for AI legal research tools that go beyond basic research capabilities include: drafting, summarision, translation, contract/document review, data extraction.
  • Which uses cases do you need? Can you select only the use cases you want from the provider, or do you have to buy into all of them?

Technical Issues

  • Deal Killer: Are research results based on “trusted content” owned by the vendor or open web sources?
  • What is the underlying AI technology? Claude, OpenAI GPT 3, GPT4. LamDa, DALLe.
  • Do they rely on multiple LLM’s depending on the use case?
  • Is the architecture model agnostic, in other words is it possible to swap one model out for another as the landscape evolves?

Research Results and Issues

  • How are you controlling for hallucinations?
  • Have clients ever experienced a hallucinated result? How was this addressed?
  • What type of testing do you do to make sure results remain reliable?
  • What is your system for notifying customers if there are bugs?
  • How does the platform respond if it doesn’t know the answer?
  • Does the result include a “confidence score”?
  • Does the answer to a research query include links to the source document?
  • Does the answer include a full citation to a sources including citation signals?
  • What is the speed of response? Do lawyers have an “email the results” option due to speed?
  • Contract /Document Review:  are there limitations on the size of the size of the document or data set that can be uploaded? Can specific clauses be analyzed?
  • Can product extract data and create a table of clauses?

 Contract Issues

  • Ask to see the license, provide it to procurement team/internal contract attorney and review it yourself. Highlight any concerns.
  • Smaller vendors may be more flexible on contract terms.
  • Does company offer “all or nothing” pricing?
  • Are you required to purchase unrelated products?
  • Liability Who is liable for false answers?
  • Are there any usage limitations e,g, caps of the number of questions or results?
  • Liability if violation client confidentiality
  • Enterprise vs. practice group vs. individual licenses
  • Length of term. One year or multiple years?
  • Product downtime
  • Notifications of system issues
  • Training support
  • Password management vs IP authentication
  • Cancellation/Autorenewal
  • Usage statistics
  • Notice of Violation
  • Opportunity to Cure
  • Copyright
  • Ongoing training

Trials – Depending Firm Culture or Ethical/Client Concerns

  • Tight control or loose control over who uses GAI at the firm or how it is used?
  • Restrict Trials to professional researchers?
  • Limit to attorneys who have a strong interest in evaluating GAI
  • Require Training on the ethical use of GAI
  • Require training on how GAI and LLMs work
  • Restrict use of GAI product to non-client work
  • Require work to be verified in a traditional legal research resource

Testing

During your testing of a product, look for:

  • Research capabilities
  • Quality of results
  • Speed of results
  • User Experience
  • Links to primary Sources
  • Citation flags
  • Editorial enhancements
  • Input -Carefully crafted Prompts
  • Input – Natural Language question like an associate assignment
  • How many “turns” in conversational Search
  • Uploading documents

Training, Deployment, Ongoing Help

  • How does the vendor support training?
  • Does the product require special prompt training?
  • Are there training guides, videos embedded in the product?
  • Is there a 24 X 7 helpline?

 Keep Learning

  • Talk to colleagues
  • Attend programs at AALL ILTA, Legal Tech
  • Set up news alerts to monitor developments.

 

[See LTH AI Legal Assistant Evaluation Framework and Requirements Template for deep dive checklists on GAI products that go beyond legal research]

About the ExpertView Profile
Jean O'Grady

Jean O'Grady

LTH Expert Venable LLP

Jean P. O’Grady has over 30 years of experience developing strategic information initiatives for Am Law 100 law firms. She holds a J.D. from Fordham University School of Law, an M.L.S. from St. John’s University and a B.A. in History from Fordham University. She is a member of the NY State Bar.

She is also a columnist at Above the Law and a panelist on Bob Ambrogi’s weekly  Legal Tech Journalist’s Roundtable.

Jean was the 2013-14 Chair of the Private Law Libraries Section of the American Association of Law Libraries and a past President of the Law Library Association of Greater New York. She is an incoming Board Member of the American Association of Law Libraries ( 2017- 2019). Board Member, New York Law Institute (2015-2018.)

She is a frequent author and speaker on the transformation of libraries and information centers, digital contract licensing , knowledge management, and the legal publishing industry. She has spoken at programs sponsored by the Information Industry Association, the Association of Newsletter Publishers, Practicing Law Institute, International Legal Technology Association , West Publishing, Price Waterhouse, LegalTech, Lexis-Nexis. Janders Dean Knowledge Management Conference, American Association of Law Schools. as well as AALL, Canadian Association of Law Libraries, Australian Law Libraries Association and the Special Libraries Association.

In 2011 she launched her blog “Dewey B Strategic” to promote awareness of the strategic importance of librarians, libraries and knowledge managers as change agents and innovators in the organizations they support. She has written provocative pieces on a variety of law firm management, publishing and technology issues which have sparked important debates in the industry. Jean started writing a regular blog for Legaltech Hub, Jean O'Grady for LTH, in 2022.

View more
Search Legaltech Jobs
Legaltech Jobs provides targeted job listings for alternative careers in law, including roles in legal technology, legal data, legal operations, legal design, and legal innovation. Click and browse to find your next opportunity!
Search Now
We use cookies to monitor the performance of our website, improve user experience, and assist in our marketing efforts. By continuing to browse our site, you agree to our use of cookies.