Where LLMs Actually Help in Business Software (and Where They Don't)

Why this question matters now

Almost every software conversation today includes the question: can we add AI to this? Sometimes the honest answer is yes, and it saves hours every week. Sometimes a well-designed form or a simple report would do the job better, faster and more cheaply.

Large language models (LLMs) are genuinely useful, but they are a specific tool with specific strengths. This guide is how we think about where they belong in business software, and where they don't.

Where LLMs genuinely help

In our experience, LLMs earn their place when the work involves language that people currently read, sort or write by hand.

• Answering questions from your own documents: policies, manuals, contracts or past tickets, with links back to the source.
• Pulling structured data out of messy inputs such as invoices, forms, emails and reports.
• Drafting first versions of routine text: replies, summaries, descriptions and notes that a person then reviews.
• Classifying and routing incoming requests so the right person sees them sooner.
• Explaining steps or concepts patiently, for example in tutoring or staff training tools.

Where they are the wrong tool

An LLM produces likely-sounding text. That makes it a poor choice wherever you need guaranteed correctness on its own.

• Exact calculations, totals and financial reconciliation. Normal code does this perfectly every time.
• Final decisions with legal, medical or financial consequences, unless a qualified person reviews them.
• Tasks where a simple rule or a dropdown already works. Adding AI there only adds cost and uncertainty.
• Anything that needs data the model has no access to. It cannot know what isn't in front of it.

What makes an AI feature reliable

The difference between a demo and a dependable feature is mostly in the engineering around the model. We focus on four things.

• Grounding: the model answers from your approved content, and shows where each answer came from.
• Permissions: users only get answers based on documents they are allowed to see.
• Evaluation: a set of real questions with known good answers, checked before every change goes live.
• Fallbacks: when the model isn't confident, it says so or hands over to a person instead of guessing.

Keeping your data private

For most businesses the first concern is data. Using providers' business APIs with data retention and training switched off, and running the surrounding system in your own cloud account, keeps your information under your control. For especially sensitive data, open-weight models can be hosted privately.

How to get started

Pick one task that happens often, takes real time and has a clear way to measure success, such as the number of support questions answered without escalation. Build a small, grounded version, test it with the people who do the work, and only then widen its scope.

If the first version doesn't clearly save time, that is useful information too. It is far cheaper to learn that from a focused pilot than from a large rollout.