A database can be UP, monitored, backed up and highly available — and still be unreliable.
That sounds contradictory.
It isn’t.
For a business-critical database, reliability is not simply about whether the database process is running.
It is about whether the database consistently delivers the service the business expects.
That distinction is becoming increasingly important as database environments become more automated, distributed and complex.
UP Does Not Mean Reliable
Imagine a monitoring dashboard showing:
- Database: UP
- CPU: 42%
- Memory: OK
- Storage: 58%
- Replication: OK
- Backups: SUCCESS
Everything is green.
The business, however, reports that the application is slow.
Customers are waiting.
Transactions take several seconds instead of milliseconds.
The database is technically available.
But from the business perspective, something is wrong.
This is one of the fundamental differences between traditional database administration and Database Reliability Engineering.
The question is no longer:
Is the database running?
The better question is:
Is the business service behaving within its expected reliability and performance boundaries?
Reliability Has Several Dimensions
When people talk about database reliability, they often immediately think about High Availability.
HA is important.
But it is only one part of the picture.
A reliable database platform must also consider:
- availability,
- performance,
- recoverability,
- data integrity,
- capacity,
- security,
- observability,
- operational consistency,
- change management,
- disaster recovery.
A database that never goes down but regularly produces severe performance regressions is not necessarily reliable.
Neither is a database with excellent performance but an untested backup strategy.
Reliability is multidimensional.
Performance Regressions Are Reliability Problems
Performance is often treated separately from reliability.
I disagree.
If an application normally processes a transaction in 100 milliseconds and suddenly requires five seconds, the service has degraded.
No server may have failed.
No process may have crashed.
No alert may have turned red.
Yet the service is no longer operating as expected.
This is why database reliability engineering must include performance engineering.
A production database should not only be monitored for failure.
It should also be monitored for behavioral change.
Execution Plans Can Change Without Anyone Intending It
One of the most common examples is SQL execution-plan change.
A query may perform perfectly for months.
Then something changes:
- statistics are refreshed,
- data distribution changes,
- an index is created,
- an index is removed,
- the database is upgraded,
- optimizer parameters change,
- the workload changes,
- the optimizer chooses another execution plan.
The SQL statement itself may not have changed.
The application may not have changed.
But the execution plan changes.
And suddenly a query that used to take 50 milliseconds takes 30 seconds.
From an application perspective, this can look like a random performance incident.
From a database perspective, it is often an optimizer and workload-management problem.
From a DBRE perspective, it is a reliability problem.
Oracle SQL Plan Management Is an Interesting Example
Oracle has provided SQL Plan Management for years as a mechanism for controlling plan stability while still allowing plan evolution.
The basic idea is important:
A database should not blindly accept every new execution plan simply because the optimizer generated it.
Instead, accepted plans can be managed and evolved based on observed performance.
Recent research around Oracle 26ai has explored extending this concept toward more immediate verification of new plans, aiming to detect regressions closer to query execution rather than relying only on background verification.
The underlying engineering principle is more important than the specific feature.
Automation should be allowed to improve the system — but changes should remain observable, measurable and recoverable.
Automation Without Guardrails Is Not Reliability
Modern database platforms increasingly automate:
- provisioning,
- backups,
- failover,
- scaling,
- patching,
- configuration,
- monitoring,
- query optimization.
Automation is essential.
But automation does not automatically create reliability.
Poorly controlled automation can simply make failures happen faster.
For example:
A configuration change is automatically deployed.
The database accepts it.
Monitoring shows the database as healthy.
Thirty minutes later, application latency increases dramatically.
Nobody immediately knows which change caused the regression.
The automation worked.
The system did not.
This is why reliable automation requires:
- observability,
- validation,
- controlled rollout,
- rollback,
- measurable success criteria.
The Database Should Have SLOs
A useful DBRE approach is to define Service Level Objectives for database-backed services.
For example:
Availability
99.99%
Transaction latency
P95 below 200 ms
Recovery Time Objective
Less than 30 minutes
Recovery Point Objective
Less than 5 minutes
Replication lag
Less than 10 seconds
Backup recovery
Successful restore test within the defined recovery window
These numbers are examples.
The correct values must come from the business.
But the principle is important.
Without measurable objectives, reliability becomes subjective.
Monitoring Should Measure User Impact
Traditional DBA monitoring often focuses on infrastructure metrics:
CPU.
Memory.
Disk.
Sessions.
Locks.
Wait events.
Replication.
These metrics remain valuable.
But DBRE monitoring should connect them to business impact.
For example:
Instead of only monitoring:
Database CPU = 80%
we should ask:
Did transaction latency increase?
Instead of:
Replication lag = 15 seconds
we should ask:
Does this violate the application’s recovery or consistency requirements?
Instead of:
Backup completed successfully
we should ask:
Can we restore the database within the required RTO?
The difference is subtle but extremely important.
A Green Backup Job Is Not Proof of Recoverability
This is one of the most common operational mistakes.
A backup system reports:
SUCCESS
That tells us that the backup process completed.
It does not prove that the business can recover.
A reliable backup strategy should periodically answer:
- Can we restore the database?
- How long does restoration take?
- Can we recover to a specific point in time?
- Are all required backup components available?
- Can the application reconnect?
- Is the recovered database consistent?
- Has the recovery procedure been tested by more than one person?
A successful backup is an event.
A successful restore is evidence.
Reliability Requires Failure Testing
The same principle applies to High Availability.
A cluster can look perfect on an architecture diagram.
Three nodes.
Automatic failover.
Replication.
Load balancing.
Everything appears redundant.
But what happens when:
- the primary server disappears?
- the network connection between nodes fails?
- storage becomes unavailable?
- replication stops?
- the standby is behind?
- DNS does not update?
- the application connection pool does not reconnect?
- the failover mechanism itself fails?
You do not know until you test it.
This is why:
HA that has never been tested is a design assumption, not demonstrated reliability.
Database Reliability Is Also About Change
Most production incidents do not happen because databases suddenly decide to fail.
Many happen after changes.
A new release.
A new index.
A configuration change.
A database upgrade.
A storage migration.
A network change.
A new workload.
A changed execution plan.
A patch.
A cloud migration.
This means that database reliability is closely connected to change management.
Every important change should have:
- a reason,
- a measurable expected outcome,
- monitoring,
- rollback criteria,
- a rollback procedure.
The objective is not to prevent change.
The objective is to make change safe.
The DBA Is Becoming a Platform Engineer
The traditional DBA skill set remains essential.
You still need to understand:
Oracle.
PostgreSQL.
SQL Server.
MySQL.
Indexes.
Transactions.
Locks.
Execution plans.
Backup and recovery.
Replication.
Storage.
Operating systems.
But modern database platforms require more.
You increasingly need:
- Infrastructure as Code,
- automation,
- observability,
- Linux,
- cloud platforms,
- CI/CD,
- configuration management,
- incident response,
- capacity planning,
- SLOs and SLIs.
This is one reason the Database Reliability Engineer role is emerging between traditional DBA, SRE and platform engineering.
The database remains the core technology.
But the responsibility expands beyond the database itself.
The Real Objective: Predictable Failure
Perfect systems do not exist.
Servers fail.
Networks fail.
Storage fails.
Software has bugs.
People make mistakes.
Cloud services have incidents.
The objective of reliability engineering is therefore not to eliminate failure.
It is to make failure:
predictable, detectable, recoverable and measurable.
That changes the engineering question.
Instead of asking:
How do we make this database impossible to fail?
Ask:
What happens when this component fails?
Then:
How quickly will we know?
Then:
What happens automatically?
Then:
What does the operator need to do?
And finally:
How do we prove that recovery works?
A Practical Database Reliability Checklist
Before calling a database platform reliable, I would ask:
Availability
Can the business service survive the failure of its primary database?
Performance
Do we know the normal performance baseline?
Recovery
Have we actually restored the database recently?
Replication
Do we measure replication lag and know its business impact?
Failover
Has automatic or manual failover been tested?
Observability
Can we detect degradation before users report it?
Changes
Can we identify which change caused a regression?
Automation
Can common operational tasks be performed consistently?
Documentation
Can another engineer recover the environment?
Ownership
Does more than one person understand the complete recovery path?
If several answers are „no“, the database may be operational.
But it is not yet a mature reliability platform.
The Most Important Database Metric May Be the One the DBA Does Not Own
The most important reliability metric is ultimately not CPU.
It is not memory.
It is not IOPS.
It is not even database availability.
It is:
Can the business service deliver what the business expects?
The database is only one component of that service.
That is why Database Reliability Engineering is broader than traditional database administration.
A DBA keeps the database healthy.
A DBRE asks whether the system around the database remains reliable under change, growth and failure.
That is a much bigger responsibility.
And increasingly, it is where the real value of senior database engineering lies.
About the Author
Tomáš Solař is a Principal Database Reliability Engineer with more than 20 years of experience working with enterprise database platforms.
His expertise includes Oracle, PostgreSQL, Microsoft SQL Server and MySQL, with a focus on Database Reliability Engineering, High Availability, Disaster Recovery, database architecture, performance engineering, automation and observability.
His approach combines traditional hands-on DBA expertise with reliability engineering and business-oriented architecture — helping organizations design database platforms not only for normal operation, but also for the moment when something inevitably fails.
Available for permanent, contract, remote and hybrid engagements.
Email: tom@tomas-solar.com
WhatsApp / Phone: +420 731 196 647
Web: tomas-solar.com