At JBF, our Strategy Practice supports clients through logistics technology selections, from defining critical requirements and developing the RFP through evaluating vendor demonstrations. During one recent selection, the client asked vendors to explain their current AI capabilities, long-term AI investment strategy, and product direction.
This was the only feature-specific question used to evaluate innovation.
The client wasn't pursuing a defined AI use case, yet AI became part of the evaluation scoring and the primary indicator of whether each provider was continuing to invest in its platform. Vendors could therefore receive favorable consideration for AI capabilities even when the use cases presented offered limited value to the client's operation. This became consequential when AI moved from a general indicator of innovation to a proposed solution for one of the client's most critical requirements.
The client's existing TMS couldn't adequately support a complex routing process, and this product gap's operational risk was one of the reasons the company had decided to pursue a new system. Simply stated, this requirement was fundamental to the evaluation.
When the use case came up during a demonstration, the vendor proposed that the client build an AI agent to support it. The client was told the agent's instructions could be defined in natural language, which suggested it could theoretically be configured to support almost any process. That breadth is hard to comprehend, let alone evaluate, because it draws no clear line between what can be imagined and what can be reliably deployed.
Given how critical the process was, we wanted to see a comparable agent function in practice and confirm that the requirement could be supported. The vendor was unable to demonstrate that capability.
This does not necessarily mean an agent was the wrong solution. I am confident, however, that it was proposed before the vendor fully understood the use case. That conclusion was reinforced by a recurring pattern throughout the demonstration, with agents consistently positioned as the solution for tasks the platform did not natively support.
The lesson from this experience extends beyond whether an agent could support one client’s specific routing use case. More fundamentally, AI may change the capabilities being evaluated, but it should not change how buyers approach a technology selection.
1. Always Start with the Operation
Before approaching the market, shippers need to understand why they're making the investment and what the technology must accomplish. That includes documenting the critical requirements a new solution must support, particularly those shaped by the company's customers, products, policies, network, and operating model.
These requirements are rarely captured through a generic feature checklist. Most shippers need routing capabilities, for instance, but the rules governing their routing decisions can be entirely different from one operation to the next. Those nuances determine whether a vendor can support the operation, regardless of whether the proposed solution relies on standard configuration, automation, optimization, or AI.
Introducing AI doesn't remove the need for that groundwork. If anything, it makes clearly defined requirements even more critical. A capability that appears highly flexible can be hard to evaluate without an established process and expected outcome to test it against.
Buyers do not need to enter a selection knowing exactly where AI should be used. They should, however, know which problems they need the technology to solve. Vendors can then explain the most appropriate way to support those requirements, whether AI is part of the answer or not.
2. The Standard of Proof Should Not Change
Earlier in my career, I worked in presales for a logistics technology provider. When a prospect needed to know whether our optimization engine could support their transportation network, we wanted to show them.
We'd request the shipper's data, model relevant scenarios, and demonstrate how the optimizer handled their operational constraints. The exercise could expose limitations, but that was part of the evaluation. The buyer needed evidence that the technology could support the business before making the investment.
If a vendor proposes AI as the solution for a critical requirement, the buyer should expect to see it applied to a comparable scenario. The demonstration will rarely replicate every nuance of the operation, but it should provide enough evidence to judge whether the approach is credible.
The same principle applies across different forms of AI. An ML prediction should be evaluated on its accuracy and usefulness within the operation. A GenAI interface should improve how users access or act on information. An AI agent should be evaluated within the process it's expected to perform, including what happens when information is incomplete or the work deviates from the expected path.
Buyers should expect proof that a proposed solution can support the requirement, regardless of what's behind it.
3. Evaluate AI Within the Flow of Work
In the TMS demonstrations I've attended, AI has often been presented separately from the end-to-end operational flow, typically through an innovation segment or standalone interaction meant to highlight what the provider is developing. While these demonstrations can be informative, they make it difficult to understand how the capability would affect daily planning and execution.
Buyers need to see AI within the process it's intended to support. A prediction becomes meaningful when the demonstration shows how it informs a decision. A recommendation becomes useful when the buyer can see how someone evaluates and acts on it. If an agent will perform the work, the demonstration should show how it handles the process and when a person has to step in.
This is the same expectation buyers should hold for any material TMS capability. They wouldn't accept that a routing guide, optimizer, or automated workflow could theoretically support their operation without seeing it function. AI should receive the same level of scrutiny.
4. Plan for Sustainable Value
An AI capability isn't a one-time implementation. Like a TMS optimizer, it may perform as intended when initially configured, but its value can deteriorate as operations change. New requirements emerge, exceptions evolve, and the instructions or decision parameters established during implementation may no longer reflect how the business operates.
Buyers need to understand the ongoing effort required before purchasing any capability, but especially advanced capabilities like AI. Who will monitor its performance, validate its decisions, adjust its instructions, and respond when the results no longer align with expectations? They should also clarify which responsibilities remain with the technology provider and which require internal resources.
The answer varies by capability and organization. IT may support the integration, security, and technical environment, while logistics operations retain responsibility for the process and its outcomes. In other cases, the provider or a centralized AI team may take a larger role. Each model carries different resource and cost implications. Vendor ownership may introduce ongoing service fees, while internal ownership may require additional capacity or skills that were not included in the original business case.
Without that clarity, the buyer is evaluating the capability at the point of implementation rather than the full commitment required to keep it useful over time.
5. Evaluate Innovation More Broadly
Buyers should keep asking vendors about AI, but that shouldn't be the only question used to assess innovation. Providers should also explain how they're improving core workflows, addressing known operational problems, and responding to changes in the logistics market. Those investments may involve AI, but they may also involve better optimization, analytics, usability, connectivity, or process automation.
Conclusion
Effective selection support goes beyond comparing feature lists and scoring vendor responses. Our role as consultants is to translate operational requirements into scenarios that vendors can demonstrate, challenge answers that remain theoretical, and confirm that the capabilities influencing the decision are reflected consistently in the proposed solution and price.
AI has considerable potential in logistics, and evaluating it well doesn't require a different process than the one that has always supported sound technology decisions. The operation still comes first. What it needs to accomplish still has to be defined before a vendor's answer can be judged, and an AI-based answer, like any other, still has to prove it can deliver before it earns a place in the decision.
In the end, AI may change what logistics technology can do, but it should not change how buyers evaluate it.
If you're evaluating an AI agent, whether a vendor is proposing it or your own team is scoping one, the harder work is defining the use case clearly enough to test it. JBF's Agentic AI Use Case Workbook offers a structured, nontechnical approach to identifying, defining, evaluating, and testing an Agentic AI use case, phase by phase, from initial opportunity through production readiness.
The results show a clear gap between AI activity and organizational readiness.
About the Author
Rachelle Butler is a Director, Strategy at JBF Consulting with more than 15 years of experience spanning logistics operations and technology. She partners with shippers to assess, design, and implement solutions that align operational needs with long-term business direction.
Rachelle’s background includes roles in product leadership, consulting, implementation, and post-deployment client success at e2open, BluJay Solutions, and LeanLogistics. She began her career in the United States Marine Corps, where she gained foundational experience in transportation coordination and logistics operations. Rachelle brings a practical, real-work approach to helping clients realize meaningful value from their operational investments.
FAQs
Only in relation to a shipper's own documented requirements, not as a stand-alone measure of innovation. In JBF's 2026 survey of 215 supply chain leaders, 78% were pursuing AI without a documented plan — exactly the gap that lets a strong AI narrative outweigh whether a platform can support the shipper's most critical process. Evaluate AI capabilities against the same requirement-driven standard used for any other feature.
Ask the vendor to demonstrate the agent against a scenario pulled from your own operation, the same standard used to test an optimization engine or routing tool. A vendor who can only describe the agent's flexibility in natural-language terms, without showing it handle a comparable case, hasn't proven it can support the requirement.
Someone needs a formal, actively maintained process for monitoring performance and adjusting it over time, and most organizations don't have one. JBF's 2026 survey found only 10.5% of supply chain leaders reported having that kind of formal post-launch governance, with ownership split across four different models and no clear majority. Buyers should settle this before signing, not after.
AI changes what capabilities a shipper might evaluate; it doesn't change the underlying process. Buyers still need to define the operational requirements a system must support before going to market, then require evidence, not a description, that a proposed AI capability can actually meet that specific requirement.
It shifts evaluation toward what's flexible and imaginable instead of what's provable. In demonstrations, agents often get positioned as the answer to almost any task the platform doesn't natively support, and that breadth is hard to evaluate because it offers no clear line between what could theoretically work and what's actually been proven to work.
