The Copilot Readiness Assessment I’d Run Before You Buy a Single Seat

We sell Microsoft licensing. So this is an awkward thing for me to write down, and I’m writing it anyway: if you buy Copilot seats this month and hand them out on Monday, you’ll waste the money, and you might create a problem you didn’t have.

Not because the product’s bad. It isn’t.

It’s what happens in the fortnight before anyone types anything.

What a Copilot licence actually does on day one

Think about a new graduate’s first morning. You’d introduce her to the team, hand her a laptop, and give her the handful of folders she needs. What you wouldn’t do, on any account, is hand her a login that opens every document in the firm.

Microsoft 365 Copilot inherits exactly the permissions the person already has. Microsoft says so plainly in its own deployment guidance: Copilot and its agents retrieve data from across your tenant and respect existing permissions and sharing settings. Nothing more, nothing less.

Which sounds reassuring, right up until you picture what your permissions look like after eleven years of trading. The folder someone shared with “everyone in the organisation” in 2019, for one afternoon. The partner drive opened up during a merger and never closed. Old payroll. A board pack. The team site nobody’s touched since the fit-out.

Nobody set out to leave any of that open. Somebody was flat out at five on a Friday, needed a colleague to see one file, and picked the sharing option that worked first go.

And nobody’s looked since, because nobody knew it was there and there was no easy way to ask.

Copilot is an extremely fast, extremely literal new starter who’ll read all of it, and answer questions about it without ever wondering whether it should.

This isn’t a theory. The Australian Government ran the experiment.

Beware: Copilot doesn’t create a permissions problem. It makes an existing one legible, instantly, to everyone in the building.

What the Commonwealth found

Between January and June 2024, the Digital Transformation Agency ran a whole-of-government Copilot trial: 5,765 licences across 56 agencies, with the evaluation published in October 2024. It’s the largest Australian evidence base we have on this, and it’s public.

One of its findings, in the report’s own words: “Use of Copilot enabled some participants to access documents that they should not have had permission to access.”

The summary report is blunter still. Copilot “may magnify poor data security and information management practices.”

If that can happen inside agencies with records teams and formal classification schemes, it can happen in a firm on Collins Street.

Two firms, same 30 days

Both boutique advisory practices. Both buy 25 seats. Both start on the first of the month.

The first firm switches everyone on that morning and sends an email with some suggested prompts. Week one is enthusiastic. By week three it’s settled into a handful of people summarising meetings. On day 30 the managing partner asks whether it was worth it, and gets a room full of genuine, unfalsifiable opinions. Somebody also mentions, a little awkwardly, that they came across a document about a former colleague’s remuneration…

The second firm spends the fortnight before day one doing three things, which together are all a Copilot readiness assessment really amounts to. They run the permission reports and find the eleven sites shared far too widely. They close them. They pick three numbers to measure, and write down today’s value for each. Then they start.

Same product, same seats, same month. One firm has a decision at the end of it. The other has a vibe and an HR conversation.

The fortnight nobody budgets for

This is the part that gets skipped, and it’s the part that decides everything.

Run the oversharing reports, and act on what they say. This is the part people mean when they say Copilot readiness assessment, and it’s less mysterious than the name suggests. The tooling lives in SharePoint Advanced Management: export your most-used sites, run a permission state report, review anything shared organisation-wide, and take back the sharing that’s no longer justified. The licensing changed here, and it’s worth knowing. It used to be a separate add-on. Now a single Copilot licence anywhere in your tenant unlocks it, so most firms are entitled to more than they realise and aren’t using it.

I’d like to tell you we were above this ourselves. We weren’t. An IT company collects the same forgotten folders as everyone else, and ours took longer to tidy than I’d have liked, which is exactly why I’d run the report before you spend a dollar on seats.

It’s permissions drift, the same thing we’ve written about with cloud settings. Copilot’s just the fastest way anyone’s yet invented to find it.

Decide what you’re measuring, in advance, in writing. Three numbers, no more. Hours a week the pilot group spends on first drafts. How many people use it on more than three days in a week. One quality measure you actually care about. Take the baseline before day one, because you can’t reconstruct it afterwards, and everyone’s memory of how long things used to take is generous.

Choose ten people, not twenty-five. Roles where the work is text-heavy and the pain is real. Then tell them plainly that the firm’s testing the tool, not testing them.

Tip: Book two 45-minute sessions in the first week, one to show people what it does and one, four days later, to hear what didn’t work. In the EY Australia survey of 1,003 workers, only 35% had received any formal AI training from their employer. That gap explains most disappointing pilots.

Then run the four weeks

One scenario a week, so people build a habit rather than a browser tab full of experiments.

• Week one, email. Catching up after a day of meetings. Summarising a long thread. Drafting a reply.
• Week two, meetings. Recaps and action items from client reviews and internal committees.
• Week three, documents. Turning file notes into a first draft. Turning a draft into a deck.
• Week four, everything else. Whatever the group has started asking for by then. That’s the week that tells you what they’ll actually pay for.

If none of those four map onto your week, go back a step and look at where AI tends to earn its keep in a firm this size. A pilot without a use case is just a subscription.

The thing to notice about the government’s numbers

Here’s what makes the Commonwealth trial genuinely useful, and it isn’t the headline.

Across the trial, 86% of participants said they wanted to keep using Copilot. 69% agreed it improved the speed of their work. Only a third used it daily, and up to 7% said it actually added time to their tasks.

Now the important part. Every one of those figures is self-reported. The evaluation’s own method appendix says the impact assessment relied on participants’ self-assessed impacts, which “may naturally under or overestimate impacts.” Participants self-nominated, and 73% were optimistic about Copilot before they started.

So the largest Copilot evaluation in the country tells us that people who volunteered, and expected to like it, liked it. That’s not nothing. It’s also not evidence of value, and the report says as much itself.

Which is exactly why your thirty days need three numbers agreed in advance. Without them you’ll land where the Commonwealth landed. A warm feeling, and no finding.

About the trial offers

There’s a partner-led trial running at the moment: 25 users, 30 days, for organisations under 300 staff, available to the end of December. That’s the route we use.

The self-serve trial on Microsoft’s website is a different animal. One month, credit card required, and it rolls into a paid annual subscription automatically unless you cancel. If you go that way, put the cancellation date in your own calendar…

One thing to do this week

Don’t buy anything yet. Ask for one report: which of your SharePoint sites are shared with everyone in the organisation, and when each of those permissions was last reviewed.

If it lands within a day, you’re in better shape than most, and you can start planning the thirty days. If it takes a week, or comes back as a conversation rather than a list, you’ve learned something more useful than any pilot would’ve told you, and it cost you nothing. Same test as the questions worth asking before you sign with any IT provider: not whether they can do it, but how fast the evidence turns up.

If you’d rather we ran that report and the clean-up behind it, that’s the sort of work our vCIO people do, and you’ll get the sites and the dates in writing. Either way, run it before you buy the seats.