Sunday, September 17, 2023

 

Resolving RPM Database Corruption Before Package Upgrades

Issue: During system maintenance on the OCI Linux environment, we encountered an issue where the RPM database became corrupted, causing dnf upgrade to fail with the following error:

 
[root@omdmz ~]# dnf upgraade
error: rpmdb: BDB0113 Thread/process 590272/140176820012928 failed: BDB1507 Thread died in Berkeley DB library

error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery

error: cannot open Packages index using db5 -  (-30973)
error: cannot open Packages database in /var/lib/rpm
Error: Error: rpmdb open failed

Step 1: Clean Up Stale RPM Database Files

To fix the issue, I first removed all temporary __db.* files:  as this clears out incomplete or orphaned transactions that may have caused the corruption.

[root@omapps01 rpm]# rm -f __db.*

D: adding 1 entries to Installtid index.
D: adding 1 entries to Sigmd5 index.
D: adding "7ab75f337467cc17322ec627c0f5525bcd83a735" to Sha1header index.
D:  read h#    2707
Header SHA256 digest: OK
Header SHA1 digest: OK
D: adding "flatpak" to Name index.
D: adding 129 entries to Basenames index.
D: adding "Unspecified" to Group index.
D: adding 78 entries to Requirename index.
D: adding 2 entries to Providename index.
D: adding 1 entries to Conflictname index.
D: adding 58 entries to Dirnames index.
D: adding 1 entries to Installtid index.
D: adding 1 entries to Sigmd5 index.
D: adding "453cd6fb88f9b42a07e8f9ab047f0e8e94e2fda8" to Sha1header index.
D: adding 2 entries to Recommendname index.
D:  read h#    2201
Header SHA256 digest: OK
Header SHA1 digest: OK
D: adding "swtpm-tools" to Name index.
D: adding 34 entries to Basenames index.
D: adding "Unspecified" to Group index.
D: adding 36 entries to Requirename index.
D: adding 3 entries to Providename index.
D: adding 15 entries to Dirnames index.
D: adding 1 entries to Installtid index.
D: adding 1 entries to Sigmd5 index.
D: adding "53d49b3bac3d0da69c55add54e90f2c33c3c5c95" to Sha1header index.
D:  read h#     667


Header SHA256 digest: OK
Header SHA1 digest: OK
D: adding "libXres" to Name index.
D: adding 8 entries to Basenames index.
D: adding "System Environment/Libraries" to Group index.
D: adding 11 entries to Requirename index.
D: adding 3 entries to Providename index.
D: adding 6 entries to Dirnames index.
D: adding 1 entries to Installtid index.
D: adding 1 entries to Sigmd5 index.
D: adding "6c7e1dc63b857095258408c570cc9ce6c15b8ec8" to Sha1header index.
D: closed   db index       /var/lib/rpm/Packages
D: closed   db environment /var/lib/rpm
D: closed   db index       /var/lib/rpmrebuilddb.1125429/Packages
D: closed   db index       /var/lib/rpmrebuilddb.1125429/Enhancename
D: closed   db index       /var/lib/rpmrebuilddb.1125429/Supplementname
D: closed   db index       /var/lib/rpmrebuilddb.1125429/Suggestname
D: closed   db index       /var/lib/rpmrebuilddb.1125429/Recommendname
D: closed   db index       /var/lib/rpmrebuilddb.1125429/Transfiletriggername
D: closed   db index       /var/lib/rpmrebuilddb.1125429/Filetriggername
D: closed   db index       /var/lib/rpmrebuilddb.1125429/Sha1header
D: closed   db index       /var/lib/rpmrebuilddb.1125429/Sigmd5
D: closed   db index       /var/lib/rpmrebuilddb.1125429/Installtid
D: closed   db index       /var/lib/rpmrebuilddb.1125429/Dirnames
D: closed   db index       /var/lib/rpmrebuilddb.1125429/Triggername
D: closed   db index       /var/lib/rpmrebuilddb.1125429/Obsoletename
D: closed   db index       /var/lib/rpmrebuilddb.1125429/Conflictname
D: closed   db index       /var/lib/rpmrebuilddb.1125429/Providename
D: closed   db index       /var/lib/rpmrebuilddb.1125429/Requirename
D: closed   db index       /var/lib/rpmrebuilddb.1125429/Group
D: closed   db index       /var/lib/rpmrebuilddb.1125429/Basenames
D: closed   db index       /var/lib/rpmrebuilddb.1125429/Name
D: closed   db environment /var/lib/rpmrebuilddb.1125429
==

Step 2:  check the location of  /var/lib/rpm 


[root@omdmz ~]# cd /var/lib/rpm
[root@omdmz rpm]# ls
Basenames     __db.001  __db.003  Enhancename      Group       Name          Packages     Recommendname  Sha1header  Suggestname     Transfiletriggername
Conflictname  __db.002  Dirnames  Filetriggername  Installtid  Obsoletename  Providename  Requirename    Sigmd5      Supplementname  Triggername
[root@omdmz rpm]#

Step 3: Rebuild the RPM db

rpm -vv --rebuilddb

This command reindexes the RPM database and ensures all package metadata is intact. The verbose flag -vv provides detailed output for troubleshooting, showing progress on database index handling.

Result

After successfully rebuilding the database, the dnf upgrade worked without errors, and the package system went back to normal.

This is a common issue on systems where RPM operations are interrupted (e.g., forced reboots, crashes). If you notice package-related errors, it’s always good practice to rebuild the RPM database after such events.