A database can be completely operational and the business can still be experiencing a serious outage.

The database process is running.

Connections are available.

CPU utilization looks normal.

Storage has plenty of free space.

The monitoring dashboard is green.

And yet users are complaining that the application is slow.

This is one of the most important differences between traditional database monitoring and modern database observability.

A database being UP does not mean that the database service is healthy.

Database Monitoring Is Not Database Observability

Traditional monitoring usually starts with simple questions:

  • Is the database running?
  • Is the server reachable?
  • Is there enough disk space?
  • Is CPU overloaded?
  • Are connections available?

These checks are necessary.

But they are not enough.

Consider a PostgreSQL database where:

  • PostgreSQL is running
  • CPU is at 30%
  • memory utilization is normal
  • disk space is sufficient
  • connections are available

Everything looks healthy.

But the application response time has increased from 200 milliseconds to 8 seconds.

From the operating system perspective, nothing may appear to be wrong.

From the business perspective, the system is already failing.

This is where observability becomes important.

What Should a Senior DBA Actually Observe?

A reliable database platform should provide visibility into several different dimensions.

Availability

Can clients connect to the database?

Performance

How long do queries take?

Capacity

Do we have enough CPU, memory, storage and connections?

Replication

Are replicas synchronized?

Recovery

Can we actually restore the database?

Workload

What is the database doing right now?

Errors

What is failing?

Trends

What is gradually getting worse?

Monitoring answers:

„Is something wrong?“

Observability should help answer:

„What is wrong, why is it wrong, and what will happen next?“

Query Latency Is More Important Than Database Uptime

One of the most dangerous monitoring mistakes is focusing too heavily on availability.

A database can have:

99.99% uptime

and still provide a terrible user experience.

Imagine an application where 1% of queries take 30 seconds.

The database may still be considered available.

But users may experience:

  • slow pages
  • failed transactions
  • timeouts
  • locked sessions
  • repeated requests
  • overloaded application servers

Eventually the problem can propagate through the entire application stack.

Database performance is therefore not simply a DBA concern.

It is a business reliability concern.

Replication Lag Is a Reliability Metric

In a replicated environment, checking whether the replica is online is not enough.

You also need to know:

How far behind is the replica?

A PostgreSQL replica may be:

UP

while being several hours behind the primary.

A MySQL replica can be:

RUNNING

while replication is effectively unusable.

An Oracle Data Guard standby can be:

AVAILABLE

while transport or apply lag is increasing.

The important metric is not simply:

Replica = UP

It is:

Replica is receiving and applying changes within the required recovery objective.

This distinction becomes critical during a failure.

Connections Can Become the Real Bottleneck

Another common problem is connection saturation.

The database itself may have plenty of CPU capacity.

But the application may be exhausting available connections.

For example:

Application
     |
Connection Pool
     |
     v
Database
     |
Maximum Connections

When the connection pool is badly configured, the database can become overloaded even when CPU utilization remains relatively low.

A useful monitoring strategy therefore tracks:

  • active connections
  • idle connections
  • connection utilization
  • connection wait time
  • rejected connections
  • connection pool saturation

A green CPU graph does not prove that the database is healthy.

Storage Latency Can Destroy Performance

Database performance is heavily dependent on storage.

A database may have:

  • sufficient IOPS
  • sufficient throughput
  • plenty of free capacity

and still experience serious performance degradation because of latency.

For databases, the difference between acceptable and unacceptable storage performance can be measured in milliseconds.

This is why a Senior DBA should not monitor only:

Disk space

but also:

Disk latency

and:

I/O wait

The question is not:

„Do we have enough disk?“

The better question is:

„Can the storage subsystem deliver the performance required by the workload?“

Locking Can Make a Healthy Database Unusable

Consider a database with:

  • 20% CPU utilization
  • normal memory usage
  • low disk utilization
  • all services running

Everything appears healthy.

But one transaction holds a critical lock.

Hundreds of other sessions are waiting.

The database is technically available.

The application is effectively unavailable.

This is why database observability needs visibility into:

  • locks
  • blocking sessions
  • waiting sessions
  • long-running transactions
  • transaction age
  • deadlocks
  • wait events

A database can have almost no CPU activity while hundreds of users are waiting for one transaction to finish.

Backups Must Be Part of Observability

Another common mistake is monitoring whether a backup job started.

That is not enough.

The real questions are:

  • Did the backup complete?
  • Was it valid?
  • Can it be read?
  • Where is it stored?
  • Is it protected from the database?
  • How old is the latest successful backup?
  • Can the database actually be restored?

A backup that cannot be restored is not a reliable recovery mechanism.

This is why recovery testing should be part of database reliability engineering.

The important metric is not:

Backup = SUCCESS

It is:

Recovery = VERIFIED

Recovery Time Must Be Measured

RTO is often documented as a business requirement.

But documentation is not proof.

Suppose the business requires:

RTO: 30 minutes

The database team should be able to demonstrate that recovery can actually be completed within that period.

That requires testing.

For example:

Failure
   ↓
Detection
   ↓
Decision
   ↓
Failover / Restore
   ↓
Database Recovery
   ↓
Application Reconnect
   ↓
Validation
   ↓
Service Restored

Every step consumes time.

If the recovery procedure has never been tested, the RTO is only an assumption.

Observability Should Follow the Business

The most important database metrics are not necessarily the ones that are easiest to collect.

A business-critical payment database may require:

  • transaction latency
  • failed transactions
  • replication lag
  • connection saturation
  • recovery status

An internal reporting database may have completely different requirements.

This means database observability should start with the question:

What does the business actually need from this database?

Then the DBA can define the appropriate reliability indicators.

From Metrics to SLOs

Modern database reliability engineering goes one step further.

Instead of simply collecting metrics, teams define measurable objectives.

For example:

Database availability

99.95%

Query latency

95% of critical queries below 500 ms

Replication lag

Less than 30 seconds

Backup freshness

Maximum 24 hours

Recovery

RTO below 30 minutes

These objectives transform database administration from:

„Everything looks green.“

into:

„The database is meeting the reliability objectives required by the business.“

That is a much stronger approach.

The Senior DBA’s Role Is Changing

A traditional DBA often focuses on individual database components.

A modern Senior DBA increasingly needs to understand the entire service.

That means connecting:

Business
   ↓
Application
   ↓
Database
   ↓
Operating System
   ↓
Storage
   ↓
Network
   ↓
Infrastructure

When a user reports:

„The application is slow.“

the DBA cannot simply check whether Oracle, PostgreSQL or MySQL is running.

The DBA needs to determine where the actual bottleneck is.

That is database reliability engineering.

The Green Dashboard Problem

One of the most dangerous situations in production is a completely green dashboard during a real incident.

It usually means one thing:

We are monitoring the wrong things.

A good observability system should allow the DBA to answer:

  1. What is failing?
  2. Who is affected?
  3. How severe is the problem?
  4. When did it start?
  5. What changed?
  6. Where is the bottleneck?
  7. Is the problem getting worse?
  8. Can we recover?
  9. How quickly can we recover?
  10. How do we prevent the same incident from happening again?

That is much more valuable than a simple:

Database UP

Database Reliability Is About Evidence

Reliability should not depend on assumptions.

We should not assume:

  • the standby will work
  • the backup will restore
  • failover will be fast
  • replication is healthy
  • storage will remain available
  • the application will reconnect
  • recovery will meet the RTO

We should test these assumptions.

Because reliability is not a configuration.

Reliability is demonstrated through evidence.

Final Thoughts

A modern database platform needs more than monitoring.

It needs observability.

Monitoring tells us that something may be wrong.

Observability helps us understand:

what happened, why it happened, who is affected, and what we should do next.

For Senior DBAs and Database Reliability Engineers, this is becoming one of the most important skills.

The goal is no longer simply:

„Keep the database running.“

The goal is:

„Understand the database service well enough to predict failure, detect degradation early, recover quickly, and continuously reduce operational risk.“

That is the difference between database administration and database reliability engineering.

About the Author

Tomas Solar
Principal Database Reliability Engineer

20+ years of experience with Oracle, PostgreSQL, MySQL, SQL Server and MongoDB.

Specialized in Database Administration, High Availability, Disaster Recovery, Performance Engineering, Database Reliability, Automation, Database Migration and Database Architecture.

The future of database administration is not about managing more servers.

It is about building database platforms that require less manual intervention and provide greater reliability.



Komentáře