AI Usage Tracking: How to Measure What AI Actually Delivers at Your Firm
Every firm deploying AI eventually gets the same question from leadership: what is it delivering? Vendor dashboards answer a different question. Here is what to track instead, who reads it, and how to count it so the number holds up.
AI usage tracking tells you who is using AI. Leadership wants to know what AI delivered. Those are different measurements. Seats, active users and prompts come from each vendor's admin console. Work completed, documents handled and hours returned come from the work itself, counted in the units your teams already use. A firm that only tracks the first can report high adoption and still not answer the managing partner, the CEO or the investment committee when they ask what the AI program is worth.
Each tool shows you its own universe
Most firms now run more than one AI tool: an assistant for everyone, something specialist for a practice or advisory team, and a growing number of agents and automations in operations. Each vendor gives you a view of its own slice. As of September 2026:
- ChatGPT Business and Enterprise list centralised admin and usage analytics (OpenAI pricing).
- Claude Enterprise lists audit logs and custom data retention (Claude Enterprise).
- Microsoft 365 Copilot points admins to Copilot Analytics (Microsoft 365 Copilot).
Those consoles are the right source for seat management, licence spend and security review. What none of them can tell you is how much of the firm's work AI is now doing, which teams it is doing it for, and what changed this week. Your tool shows you its universe. An agent shows you its universe. A leader needs the firm's.
| Usage metrics | Delivery metrics | |
|---|---|---|
| Question answered | Who is using the tool, and how often | What work AI completed, and for which team |
| Units | Seats, active users, prompts, sessions | Items, documents by type, hours, FTE capacity |
| Scope | One vendor's tool | The firm's work, across the tools it touches |
| Who reads it | IT and licence owners | Leadership, managers, compliance and security |
| Failure mode | High usage, unclear value | Only as honest as its counting rules |
Three readers, three views
One dashboard for everyone ends up useful to no one. The people who ask about AI at a firm ask different questions, and each needs the answer in a different form.
Leadership: adoption
A managing partner, CEO or COO wants a firm-wide view of all work, broken into three states: not yet automated, in progress and fully automated, team by team and week over week. That answers whether the program is moving, where it is stuck, and which teams have not started. It also turns the work still done by hand into a list someone can own.
Managers and teams: reporting
A team lead wants to quantify what their team got back, in concrete units: forms reviewed, statements filed, client folders processed, emails handled, hours returned. Not a percentage. "We automated 40% of intake" invites the question "40% of what?"; "180 new account forms reviewed this month" does not.
Compliance and security: audit logs
A chief compliance officer or CISO rarely wants a weekly summary. They want to be able to trace any agent decision when the situation demands it: what came in, which system was touched, under which permission, and what changed. A chat transcript is not that record. An AI agent audit trail is a run-by-run log a reviewer can open, follow and hand to an examiner.
What to measure
Five measures cover what leadership asks, and each one can be traced back to individual runs.
| Measure | What it counts | How it is calculated |
|---|---|---|
| Work completed | Automated runs, and the individual items processed inside each run, by business process | Count of completed runs and items |
| Documents handled | Forms, statements, client folders, tie-outs and other document types the team already tracks | Items grouped by document type |
| Time returned | Hours given back to the team | Items completed x manual minutes per item, with minutes set per process and overridable |
| Capacity created | Time returned as full-time-equivalent capacity | Hours returned in a week / 40, reported as a weekly rate |
| Adoption | Active processes, teams and functions | Counted week over week |
A worked example (illustrative)
Say an operations team automates new account form review. In one week the automation completes 200 forms. Reviewing one by hand took about 12 minutes. That is 200 x 12 = 2,400 minutes, or 40 hours returned, which is one full-time person's week of capacity. If 6 forms failed and were finished by hand, they come out of the count: 194 forms, 38.8 hours, 0.97 FTE for that week. These numbers are illustrative, not from a customer.
Notice what the report says and what it does not. It says the team got a person's week back on one process. It does not claim a salary was saved, because what the firm does with that capacity is a management decision, not an automation output. Our guide to proving AI ROI in a law firm covers turning capacity into a business case a committee will sign.
Counting rules that keep the numbers honest
- Count finished work only. Failed and cancelled runs are excluded, and an item counts once its work is complete.
- Count each week once, at close. Late-arriving items may be missed. Totals can understate; they should never overstate.
- Use per-process minutes the team can override. An estimate is a starting point. The team's own handling time is better.
- Report capacity as a rate. FTE capacity is a weekly rate; hours are what accumulate. Adding up weekly FTEs inflates the story.
- Do not count usage as value. Logins, prompts and seats belong in the licence review, not in the delivery report.
Conservative numbers are the ones that survive the follow-up question "how did you count that?". One inflated figure costs the program more credibility than a modest one ever will.
Report AI like the rest of the business
The last step is getting the numbers out of the AI tool and into the firm's own reporting. When delivery data exports into the data warehouse, AI sits in the same dashboards as revenue, headcount and client service, reviewed with the same rigour, instead of living in a vendor screenshot pasted into a slide once a quarter.
Where Caddi fits
Caddi is built around this measurement, alongside whichever assistants your firm already uses.
- Caddi Discover maps the work across the firm, the hours, tasks and documents, and ranks the most repetitive work by hours per month, so the program starts where it returns the most capacity.
- Caddi Automate, or export. High-volume, high-accuracy work runs in Caddi as deterministic code, the same way every run. Individual, judgment-heavy work can be exported to Claude as a skill or to Copilot as an agent.
- Caddi Govern measures every Caddi run and the work Caddi hands to Claude and Copilot, which reports each decision and API call back: adoption for leadership, reporting for managers, audit logs for compliance and security, and an export into your data warehouse.
What Govern does not see: a person's ad hoc chats, or agents and automations built in Claude, Copilot or other platforms without Caddi. For seat and prompt usage in those tools, the vendor's admin console is still the source. If you are still choosing an assistant, see ChatGPT vs Claude vs Copilot for lawyers.
Caddi
See how Caddi AI Agents can find your most repetitive back-office work and automate it across every tool you use
Frequently asked questions
How do I track AI usage across my firm?
Start with each vendor's admin console for seat usage: ChatGPT Business and Enterprise have usage analytics, Claude Enterprise has audit logs, and Microsoft offers Copilot Analytics. Those tell you who is logging in and how often. To know what AI delivered, track the work itself: runs and items completed, documents handled by type, hours returned per item, capacity as a weekly rate, and how many processes and teams are active. Count only finished work and exclude failed or cancelled runs.
What is the difference between AI usage and AI value?
Usage is activity: seats assigned, active users, prompts sent. Value is work done: forms reviewed, statements filed, hours given back to a team. A firm can have high usage and little to show for it, or a small number of automations that return a full person's week. Leadership asks about value, so report work completed in the units teams already use.
Which AI adoption metrics should a COO report?
Five cover most of it: work completed (runs and the items inside them), documents handled by type, time returned (items multiplied by the manual minutes per item), capacity created (hours returned divided by a 40-hour week, as a weekly rate) and adoption (active processes, teams and functions, week over week). Add the share of the firm's work that is fully automated, in progress and still manual, by team.
How do you calculate hours saved by AI without inflating them?
Multiply completed items by the manual minutes one item takes, using a per-process estimate the team can override with its own handling times. Exclude failed and cancelled runs, count an item only once its work is complete, and count each week once at close. Report capacity as a weekly rate rather than a running total. Conservative numbers survive the question 'how did you count that'.
Do I need an audit trail for AI agents?
If an agent touches client records, compliance and security will eventually ask what it did and why. An AI agent audit trail should let them trace any run decision by decision: what came in, which system it touched, under which permission, and what it changed. Conversation history in a chat tool is not the same thing as a run record.
Can Caddi track usage in ChatGPT, Claude or Copilot?
Caddi Govern measures every Caddi run and the work Caddi hands to Claude as a skill or to Copilot as an agent, which reports each decision and API call back. It does not see a person's ad hoc chats, or automations built in Claude, Copilot or other platforms without Caddi. For seat and prompt usage in those tools, each vendor's admin console remains the source.