What LLMs are useful for—and when not to use one

LLMs are strongest when language is the interface, several answers could be acceptable and mistakes can be detected before they cause harm. They are weakest when a system needs exact state, deterministic behaviour or unreviewed authority.

The easiest way to waste an LLM is to give it work that ordinary software already performs better.

A team replaces a database filter with a chat interface. The answer becomes slower, more expensive and less predictable.

Another team asks a model to draft incident summaries from several logs. Operators review the drafts, correct missing details and save hours of routine synthesis.

Both teams are “using AI.”

Only one matched the tool to the work.

The central test

Use an LLM where interpretation and language variation create value. Use deterministic systems where correctness is already expressible as rules, queries or calculations.

LLMs are good at transforming language

The most reliable LLM tasks often begin with existing material and ask for a transformation.

Examples include:

summarising a long document
rewriting technical text for another audience
extracting proposed actions from meeting notes
comparing two policies
classifying support requests
converting prose into a draft structure
explaining code
producing an initial test case
generating search terms
translating between natural language and a formal format

These tasks constrain the model.

The source material is present. The expected output can be reviewed. The model is not required to invent an unknown fact.

A useful mental model is:

Source material
      |
      v
Language transformation
      |
      v
Reviewable output

The model still may omit, distort or overstate something.

But a reviewer has an object against which the output can be compared.

This is safer than asking the model to answer an open factual question from parameter memory alone.

LLMs are useful interfaces to complex information

Many organisations have information that is technically available but difficult to use.

Documents sit across several repositories. Policies use specialist language. Infrastructure records require knowledge of several tools. Employees know that an answer exists somewhere but not where.

An LLM can provide a natural-language layer over that complexity.

The model can:

interpret the user’s question
generate several search formulations
retrieve relevant records
explain the result
identify unresolved gaps
present the answer at an appropriate level

The value does not come from replacing the source systems.

It comes from reducing the effort required to navigate them.

The source systems should remain authoritative.

An AI assistant may explain a storage policy. It should not silently become the master copy of that policy.

Drafting is valuable when the first version is expensive

Many organisational tasks are not difficult because every sentence requires rare expertise.

They are difficult because a blank page is costly.

An LLM can produce a first structure for:

project briefs
architecture decision records
change communications
operational procedures
training material
procurement questions
risk registers
code documentation

A skilled person can then spend time on the decisions rather than on creating the initial shape.

The benefit depends on review.

A generic draft that is published with minimal scrutiny transfers the model’s omissions into organisational documentation. A draft reviewed by the person accountable for the decision can reduce effort without transferring authority.

Useful division of work

Let the model accelerate expression. Keep responsibility for meaning with the person or team that owns the result.

Coding assistance is strongest inside an engineering system

LLMs can write functions, explain errors, generate tests and translate between programming languages.

They are particularly useful when:

the required behaviour is clearly specified
automated tests exist
code review is mandatory
dependencies are controlled
generated code runs in an isolated environment
security-sensitive changes receive specialist review

The model can produce syntactically plausible code that is insecure, inefficient or subtly wrong.

That is not unusual behaviour for a language model. Code is another language sequence, and several continuations may look plausible.

The surrounding engineering controls turn generated code into usable software:

Requirement
    |
Generated change
    |
Static checks
    |
Tests
    |
Human review
    |
Controlled release

Without that path, faster code generation can simply create a larger review backlog.

Reasoning models improve some tasks, not every task

Models trained to spend more computation on difficult questions have improved performance on several mathematical, coding and other verifiable benchmarks.

The DeepSeek-R1 research, for example, reported gains from reinforcement learning and longer reasoning behaviour, especially where answers could be checked through objective rewards. It also described limitations and did not establish equivalent reliability for subjective or open-ended decisions. (arXiv)

This distinction is important.

A mathematical answer can often be checked.

A hiring recommendation, policy judgement or architecture decision may have no single correct result. Longer generated reasoning can make the answer more detailed without making the underlying assumptions better.

The correct response to a “reasoning model” is not to remove review.

It is to design a stronger verification method for the task.

Some tasks should remain ordinary software

An LLM is usually the wrong primary tool when the task requires:

Exact arithmetic

Use a calculator, database or numerical library.

The model may explain the calculation, but the deterministic tool should perform it.

Authoritative record lookup

Use the source system.

The model can help formulate the query or explain the result. It should not invent a record that does not exist.

Stable business rules

Use a rules engine or application logic when the rule can be written explicitly.

A model should not decide whether an invoice exceeds a fixed threshold through free-form interpretation.

Safety-critical control

Use engineered control systems with known timing and failure behaviour.

A model may support diagnosis or documentation. It should not become an undeclared control loop.

High-impact decisions without review

Employment, credit, medical, legal and access decisions can materially affect people.

A fluent recommendation does not remove the need for lawful process, qualified judgement, traceable evidence and appeal.

Repeated high-volume work with little ambiguity

A deterministic parser or classifier may be faster, cheaper and easier to operate after the pattern is understood.

LLMs are useful during discovery. The mature production solution may be smaller and simpler.

Risk depends on consequence, not cleverness

A model writing alternative headlines has a different risk profile from a model recommending whether a person receives a service.

A practical classification is:

Task class	Example	Appropriate control
Assistive	Rewrite a paragraph	User review
Informational	Answer from internal documents	Citations and source checking
Analytical	Compare architecture options	Expert review and explicit assumptions
Transactional	Create a ticket or draft change	Scoped permissions and confirmation
Consequential	Affect rights, safety or material access	Formal governance, evidence and accountable human authority
Autonomous	Continue acting across systems	Strong identity, budgets, monitoring and stop controls

The model may be identical across all six classes.

The infrastructure should not be.

Human review must be designed, not merely mentioned

“Human in the loop” is often written into an AI proposal as though it solves every risk.

It does not.

A reviewer can become a rubber stamp when:

output volume is too high
the model is usually correct
the reviewer lacks subject expertise
the source material is unavailable
deadlines discourage investigation
the interface presents the answer as authoritative

Review needs a practical contract.

Who reviews? What evidence can they see? Which failures should they look for? Can they reject the output? Is there enough time? Is disagreement recorded?

For low-risk drafting, ordinary user judgement may be enough.

For high-impact use, review must be part of the operating design.

Evaluate the task, not the model’s personality

A general conversation with a model is a poor evaluation.

The model may perform impressively on ten questions and fail on the eleventh that matters.

Build an evaluation set from real work:

representative inputs
difficult edge cases
incomplete information
adversarial instructions
sensitive data conditions
expected refusals
known historical failures

Then define what counts as acceptable.

Possible measures include:

factual support
completeness
classification accuracy
valid output structure
correct tool choice
unsupported-claim rate
human correction time
latency
cost
failure severity

The evaluation should compare the LLM with the current process.

A model that reaches an attractive benchmark score may still be worse than a simple template for the organisation’s actual task.

NIST’s Generative AI Profile places testing, evaluation and risk management across the AI lifecycle rather than treating model selection as the final governance step. (NIST)

The benefits and costs belong in the same decision

Benefit	Corresponding cost or risk
Flexible language interface	Variable and sometimes incorrect interpretation
Fast first drafts	More material requiring review
Broad task coverage	Harder testing across every task
Natural-language search	Retrieval and citation failures
Code generation	Security and maintenance burden
Tool use	New permissions and attack paths
Rapid experimentation	Prototypes entering production without operating controls
Model choice	Version, licence and provider dependencies

The correct question is not whether the benefits are real.

They are.

The question is whether the organisation has designed for the corresponding costs.

Start with one bounded problem

The safest adoption pattern is narrow:

choose one real problem
define one useful outcome
identify the authoritative data
establish a non-AI baseline
build an evaluation set
keep authority bounded
observe actual use
expand only when evidence supports expansion

This follows the same discipline LibreInfra applies to its editorial and architecture work: one topic, one deep and inspectable unit, rather than a bundle of vague claims.

A broad “AI assistant for everything” is difficult to evaluate and govern.

A system that drafts maintenance summaries from approved records has a clear boundary.

Useful AI begins with a specific responsibility.

One question for the next AI proposal

Do not ask whether the model can perform the demonstration.

Ask:

What is the cheapest, most deterministic system that could solve this problem—and what additional value justifies introducing an LLM?