AI agents are starting to take on more of the work inside IT operations, from investigating issues to initiating remediation. That should remove repetitive work from IT teams and get employees back to work faster. But as agents move into production, I think we need to be much more demanding about how we measure their impact.
Automations completed and tickets deflected have been useful measures of efficiency, but they are no longer enough on their own. An agent can successfully remediate an issue without removing the reason it happened. It can make resolution faster without necessarily making the environment more reliable. And if we stop measuring once the agent finishes its task, we don't know whether the fix actually held.
For me, that's where measuring AI in IT gets more interesting. The activity matters, but so does what changed because of it.
Successful remediation can still hide a recurring problem
An IT team’s view of technology can be very different from the employee’s. A network can be available, an application can be running and service levels can be met while somebody is still unable to do what they need to do. Anyone who has sat through a video call where everything is technically “up” but the experience is terrible will recognise the distinction.
This is the gap that digital employee experience (DEX) solutions are designed to expose. DEX uses evidence from the devices, applications and services employees rely on to understand how workplace technology is actually performing where the work happens. For an AI agent, that gives another important source of truth alongside historical tickets, knowledge articles and infrastructure data. The history matters, but so does the context around the employee and their technology when the problem occurs. An agent needs both if it is going to understand what it is dealing with rather than simply recognise something that looks like a problem it has seen before. The same applies to remediation. Real-time response is valuable, and responding in milliseconds is clearly preferable to leaving an employee struggling. But a millisecond reaction is still a reaction. When the same condition keeps triggering the same remediation, the speed of the fix is no longer the only interesting thing about it.
Repeated automation isn’t necessarily a problem. There will be cases where automating a known condition makes more sense than trying to engineer it out of existence. But if the same remediation keeps firing, it is worth understanding why. It could point toward an application, configuration, update or another underlying problem that deserves attention.
At a large hospital network, a small number of recurring issues were driving a disproportionate amount of service desk activity. These included problems with file synchronisation, security services, CPU utilisation, network services and collaboration tools. Addressing those recurring causes reduced annual ticket volume by roughly 50,000 incidents, with estimated savings of approximately $1.6 million.
Investigate resolution time
The remediation itself is only one part of resolving an IT problem. A surprising amount of time can disappear before anybody gets near the fix.
In my experience, as much as 60 to 70 per cent of ticket resolution time can be consumed by the interrogation around a problem. This can mean establishing who the user is, which device they are using, what they are entitled to access, what has changed and what was running when the problem occurred. When a ticket moves to a higher support tier, those same questions are often asked again.
I think of that as a verification tax. The information is necessary, but repeatedly reconstructing the context around a problem takes time. This is an area where agents have the potential to make a very practical difference. An agent that can assemble reliable context from the environment, correlate what changed and provide the evidence behind its diagnosis can take some of that repetitive investigation away from engineers.
The quality of that context matters as much as the speed with which an agent can assemble it. Historical tickets and knowledge bases can tell an agent how similar problems were resolved in the past, and that is valuable information. But IT problems do not always repeat themselves neatly. Two incidents can look similar from a ticket description while the state of the device, application or network is quite different. Giving an agent access to current endpoint evidence alongside that history makes it much easier to establish what is actually happening now, rather than assuming that yesterday’s answer still applies.
PIMCO offers a useful illustration. The global investment management firm’s infrastructure team used endpoint evidence and change correlation to reduce root-cause investigations that had previously taken days to minutes.
For AI agents, that makes the work surrounding the fix worth watching. Resolution may become faster, but where did the time come out? If it came out of repeated diagnostics, context gathering and investigation, that is meaningful operational effort being removed.
Check whether the fix holds
The more we allow agents to remediate autonomously, the more important validation becomes. A successful action is one thing; knowing that it solved the problem and that the fix held is another.
This is where DEX data becomes particularly useful. As agents take on more remediation, IT should be aiming for much greater depth and frequency in its endpoint telemetry. Being able to draw on thousands of data points from each endpoint, refreshed every few seconds, gives IT a much clearer picture of conditions before and after an intervention. The point isn't to collect more data for its own sake; it's to have enough evidence to know whether the technology actually recovered and whether the improvement held.
If performance returned to its normal range or an application stopped crashing, the data should show it. Equally, if the same condition starts to deteriorate again, IT should be able to see that too. Validation doesn’t require putting a human approval into every automated workflow. That would undermine much of the speed and scale we want from autonomous operations. It means having enough evidence to know what happened after the intervention.
Service desk figures can miss something else here. Employees are remarkably good at adapting to unreliable technology. They reboot, relaunch applications and retry failed tasks. Once somebody knows the workaround, they may stop raising a ticket altogether. From the service desk, demand has fallen. From the employee’s perspective, the problem is still part of the working day.
That is why the evidence after an intervention matters. If the agent has genuinely improved the employee’s experience, IT should be able to see that in the technology itself, rather than inferring success from a closed ticket or completed automation.
AI agents should take a great deal of repetitive work out of IT operations. But I don’t think we should confuse more automated activity with better IT. The real value is in what no longer needs fixing, investigating or working around.