In existing operational environments, there are core processes in flight. Often an initiative will start with a key bottleneck identified by office staff and a lead employee’s description kicks off the investigation. That person should participate in the initiative, typically as a user stakeholder and subject matter expert. In the case of Morty, a broker fulfilled these roles.

Existing operations also provide a non-AI baseline, which is useful. For Morty, the non-AI baseline was human search within a PDF collection. The regulations are so dense that staff spent time building additional PDF indices, another baseline component.

Depending on how many iterations have happened already, you may have key documents to draw from. This is common in larger firms where operational initiatives are more formal. If they exist, use them.

After analysis, identified problems were:

• Large amount of staff time spent searching
• Limitations in search itself: Had to know keywords, secondary index was created
• Regulatory updates set back the process: Index needed update, files had to be reliably distributed

Therefore, the project began with these goals:

• Reduce research time
• Improve search experience
• Remove indexing and reduce distribution effort

How could AI help? Working with the lead user, I made a no-code prototype using Make.com. Prototyping is not a required step. However, a chat-style tool was a natural fit for the problem and could be quickly prototyped. The tangible landmark helped non-technical stakeholders describe what they wanted and understand what they could get.

We soon agreed that an LLM chatbot was an improvement over a folder of PDFs: Quicker retrieval and an easier UI. Having that validation of user experience, we defined the preferred approach: Mortgage brokers making a request using a Discord chatbot to an LLM that searches a vectorized knowledge base. Bot responses would have to be “conversationally fast”. The knowledge base and chatbot remove the indexing and distribution effort.

A successful tech adoption requires more than just a tech shopping list though. Using the systems thinking rubric of people, processes, and technology, we considered these questions:

People

• Who will create, monitor, and maintain the AI tools?
• Whose daily life is impacted by the change, including indirectly?
• Who will use it directly, and what training will they need?
• Who is the business owner who will put this into production?
• Who will own it day to day once in production?
• Who will be the first to experience/observe a failure?

Processes

• Is the entire process changed or just a part?
• When/Where/How are failures felt?
• How is training delivered?
• How will updates be handled?
• What’s the human-in-the-loop plan when the AI system is wrong?

Technology

• What are the technical elements, including non-AI ones?
• What are the expectations regarding result quality, data security, explainability, and transparency?

With Morty, the Discord bot and application hosting were non-AI elements but just as important as the LLM and vector database. Be sure not to miss non-AI elements in the excitement of adopting new technology.

Regarding result quality, a common approach is to classify answers on a “confusion matrix”: A grid with the dimensions of true/false and positive/negative. In the case of Morty, a key issue was false positives, or incorrect answers presented as correct (hallucinations). For a mortgage broker, a false positive is a rabbit hole that leads to rework. A false negative, on the other hand, is an incorrect “I don’t know” that simply means rephrasing a question or, at worst, reverting to the baseline PDF—undesired but not problematic.

Keeping personal information out of the chat was also important—the chatbot regularly posted a reminder to not do so. Additionally, you will want to set a house standard for AI explainability and transparency that meets regulatory requirements and the voluntary standards you have chosen to adopt.

As an initiative champion, you have expertise in best practice goals and metrics setting. They are as relevant here as in other initiatives; cost-benefit and ROI analysis skills will be valuable. Additionally, you will need to define acceptable AI system performance.

First, the problem statement needs to be refined into specifics that can be actioned. Although the initiative can be summarized as “create a research chatbot”, that is not a success criterion.

Morty’s success criteria measure the bot’s time to respond and its false positive rate: “Provide a research response within 1 minute, with employee rejection of <5% of responses.” In this case, we used “1 minute” as an initial target to operationalize “conversationally fast”, turning an aspiration into a number. Both that number and the 5% are expected to change as we refine our work. The benefit is that now we have numbers to aim at.

In a small office, this is sufficient and measurable with simple chat logs and employee interviews. In a large branch office situation, we could expand the criteria to link to sales funnel velocity held in a CRM. Your specific context influences the process.

If you are about to start an AI project or the next iteration of one, review the people, process, and technology questions above. Then, attempt to write out:

• One sentence naming the problem and who has it.
• The non-AI option/baseline and why it falls short.
• One measurable success criterion.

With that, you and your CPMAI are off to a good start.