Navigating the AI implementation journey involves overcoming challenges that you never expected to face. We’re in the middle of a major project to implement AI in our business. Our goal is to train a Large Language Model on all our documentation and knowledge bases, so that our team can ask questions through Microsoft Teams about our business and get instant, sourced answers by an AI chatbot. Here are the unexpected challenges we’ve run into:
4 Unexpected AI Challenges
Deciphering Vocabulary and Context
One significant challenge was interpreting nuanced vocabulary within questions, which varied based on the asker’s role. Overcoming this involved teaching the AI to recognize abbreviations, understand conversational cues, and ask clarifying questions in the face of vagueness.
The system needs to:
- Recognize abbreviations.
- Understand conversational cues.
- Ask clarifying questions when faced with vagueness.
For instance, understanding the meaning of terms like “our” and “we” required training the AI to adapt to different perspectives. Additionally, dealing with client names, abbreviations, and ensuring accurate identification posed a substantial challenge.
Managing Unrealistic Expectations
Another challenge revolved around unrealistic expectations about AI capabilities. Addressing the misconception that “AI can do anything” and dealing with tribal knowledge references proved essential. It became apparent that users experimenting with AI for fun had different expectations than those seeking to accomplish meaningful work. Managing these expectations early on became crucial for building user confidence.
“AI can do anything”:
If you provide a tool with an open, blank text box for an interface, everyone assumes they can be very creative with what the tool can do.
Tribal Knowledge:
Shorthand and tribal knowledge references are natural for people but very difficult for AI.
Work vs. Play:
Many, if not most people have experimented with AI for fun. Trying to interact with an AI system to accomplish meaningful work versus a real deadline changes the feel of it.
Giving up:
If the tool doesn’t live up to each user’s expectations within the first few queries, they won’t have the confidence to invest more effort to use it later.
Data & Skills Refinement
Refining both skills and information storage was a necessity, driven by insights from developers expecting seamless integration with our IT service management system. Contrary to expectations, the platform’s customizable nature posed challenges in obtaining structured and consistent information.
Teaching the AI to comprehend information intricacies, draw inferences, and discern authoritative knowledge sources became imperative for successful integration.
Adoption
Two months post-launch, we faced the reality that less than 10% of our service team actively engages with Neo on any given day.
With approximately 120 team members, we currently only witness:
- Approximately 12 unique users daily
- 10 to 35 conversation sessions
Recognizing the gap between expectations and reality, we are actively working on course adjustments to bring Neo closer to our team’s daily workflows. This introspective approach acknowledges the ongoing effort and belief required for successful AI adoption.
Lessons Learned:
- Up-Front Buy-In and Commitment: Recognizing the importance of upfront buy-in from intended adopters is crucial. Simply handing an AI tool to someone doesn’t guarantee transformative results; individuals must be personally invested in adopting new habits for meaningful change.
- AI as an Information Quality Tool: AI excels in illuminating the quality, conflicts, and depth of information sources. Gaps in the AI system become apparent when there’s no human interpretation, emphasizing the need for a human touch in navigating data nuances. Leveraging AI to enhance information quality becomes a valuable aspect of the implementation journey.
- Integration into Existing Workflow: Integrating AI directly into existing tools and workflows is more effective than expecting users to adopt new work habits. While large language models excel in short to midterm applications with broad knowledge, specialized business and industry contexts require extensive effort in integrating skills and vocabulary for consistent, meaningful outputs.
- AI Adoption: Acknowledging the ongoing nature of AI adoption, continuous effort, belief, and refinement are essential to align with organizational goals and expectations.
Approaching the implementation journey with strategic planning, commitment, and a clear understanding of AI’s role enhances existing processes. While challenges persist, positive strides with around 10% of our service team using Neo daily underscore the need for ongoing adjustments to seamlessly integrate AI into daily workflows for sustained efficiency.
Watch the Full Webinar
Aldridge CIO, Chad Hiatt gives a candid update on our project to implement AI across our business. We want to share our AI journey, warts and all, because we believe that everyone who is interested in AI will benefit from the lessons that we have learned the hard way. Watch the full 2024 AI Update webinar to see the lessons learned from our AI journey.