Apress.Pro.Ubuntu.Server.Administration

CHAPTER 13 N

340 CHAPTER 13 N

CHAPTER 13 N

C H A P T E R 14

Ubuntu Server Troubleshooting
Fixing the Most Common
Problems

Although Ubuntu Server is an extremely stable server operating system, you might

encounter problems occasionally, ranging from a Linux-related issue to a simple hard-
ware failure. In this chapter you’ll learn how to troubleshoot some of the most common
problems.

Some say troubleshooting is difficult and requires years of experience. Experience
indeed helps, but a good analytical mind is the most important troubleshooting tool. In
day-to-day troubleshooting, you first have to determine where exactly a given problem
has occurred. If, for example, you have a problem with a kernel module, it doesn’t make
much sense to troubleshoot your web server.

After determining the location and scope of the problem as well as you can, you can
apply your skills to fix the problem. This requires that you have a good understanding of
how the erratic system component is supposed to function and can choose the correct
tool to repair it. This chapter first explains how to determine where exactly a problem has
occurred. Next, it introduces you to some of the best troubleshooting tools to use. Finally,
this chapter identifies some of the most common problems and explains how to fix them.

NNote This chapter assumes that you are familiar with basic principles of Ubuntu system administration.

If you want to refresh your knowledge, try Beginning Ubuntu LTS Server Administration, Second Edition, in
which I explain essential concepts such as the boot procedure and kernel management.

343

344 CHAPTER 14 N

CHAPTER 14 N

346 CHAPTER 14 N

CHAPTER 14 N

348 CHAPTER 14 N

CHAPTER 14 N

350 CHAPTER 14 N

CHAPTER 14 N

352 CHAPTER 14 N

CHAPTER 14 N

354 CHAPTER 14 N

CHAPTER 14 N

356 CHAPTER 14 N

CHAPTER 14 N

358 CHAPTER 14 N

CHAPTER 14 N

360 CHAPTER 14 N

CHAPTER 14 N

362 CHAPTER 14 N

CHAPTER 14 N

364 CHAPTER 14 N

CHAPTER 14 N

366 CHAPTER 14 N

CHAPTER 14 N

368 CHAPTER 14 N

CHAPTER 14 N

370 CHAPTER 14 N

CHAPTER 14 N

372 CHAPTER 14 N

CHAPTER 14 N

374 CHAPTER 14 N

CHAPTER 14 N

376 CHAPTER 14 N

CHAPTER 14 N

378 CHAPTER 14 N

CHAPTER 14 N

380 CHAPTER 14 N

CHAPTER 14 N

Index

Symbols monitoring status of, 299–302
overview of, 290
64-bit version of Ubuntu Server, 7 permissions, 293
updating profiles, 299
A apparmor status command, 299
apt command, 31
aa-complain command, 299 apt-get command, 163
accepting mail from servers on Internet, apt-get install drbl command, 31
apt-get purge slapd command, 205
267–268 apt-get update command, 163
accessing damaged Ext3 file system, aptitude search samba command, 232
assigning directory for Clonezilla, 35–36
378–380 attributes of LDAP objects, 198
access lookup table, configuring Postfix to authentication
See also Kerberos
use, 270 for enterprise network installation, 3
ACLs, slapd.conf file, 214–215 of Linux users, 245–246
Active Directory and Samba using LDAP for

Kerberos authentication, setting up, 245 nsswitch.conf file, 228–229
making Samba member of domain, 244 PAM, configuring, 223–228
overview of, 241, 244 testing client connectivity, 230
active memory, 58 authkeys file, 176
administrative user account for Kerberos
server, creating, 337–338 B
administrator password, lost, dealing with,
backing up
380–381 cluster configuration, 187–191
aliases lookup table, configuring Postfix to files, snapshot technology for, 5

use, 273 balanced trees (b-trees), 115
allocating files in file systems, 111 barrier option (XFS), 127
allocation groups, 112, 125 barriers, to avoid reordering in journal file
allocsize option (XFS), 126
analyzing Ext2/Ext3 file system systems, 114
Beginning Ubuntu Server Administration
debugfs utility, 122–124
dumpe2fs and tune2fs utilities, 120–122 (van Vugt), 131
e2fsck utility, 120 bitmaps, inode and block, 112–113
Anticipatory scheduler (I/O scheduler), 97 blocked process, 51
Apache Web Server, high-availability solu- boot partition, creating, 12
boot problems of LVM logical volumes,
tion for, 194
AppArmor 368–373

components of, 290–291
creating and managing profiles,

294–299
installing and starting, 293–294

383

384 NINDEX

boot process, phases in, 344–350 installing software, 31–32
/boot/grub/menu.lst file, 351 setting up, 30
setting up, 29
C cluster configuration, backing up, 187–191
cluster resources, configuring, 180–187
caching, types of, 91 commands
canonical lookup table, configuring Post- aa-complain, 299
fix to use, 270 apparmor status, 299
certificate authority (CA) apt, 31
apt-get, 163
creating, 284–289 apt-get install drbl, 31
need for, 283–284 apt-get update, 163
OpenVPN, configuring, 305–308 aptitude search samba, 232
cfg_file statements (Nagios), 136 check, 148–149
changing password in LDAP environment, check-nrpe, 154
cibadmin, 189–191
223 cibadmin -Q, 177, 179, 187
check command (Nagios), 148–149 crm_mon -i 1, 177, 183
check_disk plug-in (Nagios), output from, debugfs utility
help, 124
144–147 ls, 122
checking NTP synchronization status, lsdel, 123
dpkg-reconfigure postfix, 257–262
326–327 fdisk, 366–367
check_nrpe command (Nagios), 154 fdisk -l, 359
chroot environment free -m, 57
gpart, 365–366
activating from Knoppix Rescue CD, help, 326
357 ietadm, 171
ifconfig, 73–74
listing, 360 ipcs -m, 95
troubleshooting and, 357 iscsiadm, 173
cibadmin command, 189–191 iscsiadm -m session, 174
cibadmin -Q command, 177, 179, 187 kadmin.local, 337, 338
cib.xml file, 177–179 kinit, 338
cleanup daemon, 251 klist, 338
client keys (OpenVPN) LDAP
copying, 312 apt-get purge slapd, 205
creating, 310–311 dpkg-reconfigure slapd, 203–206
clients for management, 220–221
See also Kerberos client getent, 230
cloning, 39–43 ldapadd, 201
configuring for cloning, 36–38 ldapdelete, 201, 222
NTP, configuring, 325–326 ldapmodify, 201, 221
Clonezilla imaging server ldappasswd, 223
assigning directory for, 35 ldapsearch, 201, 217–220
cloning clients, 39–43 ldd, 223
configuring, 36
configuring for cloning, 36–38
Diskless Remote Boot in Linux

configuring software, 32–33
DHCP server, setting up, 33–35

NI N D E X 385

ls -i, 110 xfs_growfs, 125–127
lsof, 71 xfs_info, 125
lsscsi, 174 xfs_repair, 127
lvchange, 374 xfs_rtcp, 127
lvdisplay, 369–373 common-auth configuration file (PAM)
lvscan, 371–373 contents of, 227
mkdir /share, 232 modified, 228
mount, 235 Complete Fair Queueing (I/O scheduler),
Nagios tool, 135, 148–149, 154
netstat -tulpn, 78 97
nice, adjusting process priority with, components

89–90 Cyrus IMAPd, 275–276
ntpdate, 325 Postfix MTA, 262–263
ntpq, 326 configuration file
ntpq -p, 326 storing settings in, 84
ntptrace, 326 configuration files
OpenLDAP, 201 for Nagios tool
openssl, 284
openvpn, 314 contacts.cfg, 139
/opt/drbl/sbin/drbl-client-switch, contacts group, defining, 140
creating, 138–139
37–38 hosts and host groups, defining,
/opt/drbl/sbin/drblpush -i, 32
/opt/drbl/sbin/select-in-client, 38 141–143
pmap, 63, 65 services.cfg, 144–148
ps aux, 62 timeperiods_nagios2.cfg, 149–151
postdrop, 251 OpenLDAP, 201
pvdisplay, 369 configuring
pvscan, 368 See also configuration files
reiserfsck, 380 clients for cloning in Clonezilla imaging
smbclient, 235
smbclient -L localhost, 234 server, 36–38
smbpasswd, 231, 235 Clonezilla imaging server, 35–36
smbpasswd -W, 239 cluster resources, 180–187
stat, 110 Cyrus IMAPd, 276–277
stats, 112 Diskless Remote Boot in Linux software,
sysctl, 84, 85
taskset, 90, 91 32–33
time, 80 DRBD, 164–165
unconfined, 301 Heartbeat, 175–179
vgdisplay, 370 huge pages, 92–93
vgscan, 370 iSCSI initiator, 173–175
vmstat -d, 67 iSCSI target, 169–172
vmstat -s Kerberos, 330–332
Kerberos client, 339
CPU performance and, 52, 58 Kerberos server
memory performance and, 59
xfsdump, 127 administrative user account, 337–338
xfs_freeze, 128 database and stash file, 336
generic settings, 332–335
KDC settings, 335–336
user accounts, adding, 338
verifying KDC is operational, 338–339

386 NINDEX

Nagios server to use NRPE, 154–155 as primary domain controllers,
Nagios tool 241–243

location of configuration files, testing access to share, 235–236
135–136 user account, creating, 235
workstation accounts, creating, 243
master configuration file, 136–138 stand-alone NTP time server, 323–324
restarting with configuration, 151 STONITH, 191–193
NRPE on monitored server, 152–154 Ubuntu Server as mail server, 249
nsswitch.conf file to find LDAP services, VPN server
Linux VPN client, 316–319
228–229 overview of, 313–316
NTP client, 325–326 Windows VPN client, 320
NTP time server connection to SAN for enterprise network

drift file, 327 installation, 2–3
log file, 328 contacts file for Nagios tool, creating, 139
security restrictions, applying, contacts group for Nagios tool, defining,

328–329 140
OpenLDAP container objects, 198
contents of inodes, showing, 110–111
database, adding information to, context switches, performance and, 52
215–217 copying keys to client (OpenVPN), 312
CPU
overview of, 202
server, 203–215 analyzing performance of, 51– 57
slapd server, 203–206 monitoring with top utility, 46– 49
verifying configuration with tuning

ldapsearch command, 217–220 adjusting process priority using nice
OpenVPN command, 89–90

certificate authority, 305–308 overview of, 87– 89
client keys, 310–311 SMP environments, 90–91
copying keys to client, 312 thread scheduler, 87–88
Diffie-Hellman parameters, 311–312 crm_mon -i 1 command, 177, 183–184
server keys, 308–310 cryptography
PAM for LDAP authentication, 223–228 certificate authority
Postfix MTA creating, 284–289
components, 262–263 need for, 283–284
dpkg-reconfigure command, 257–262 key pairs, 282
global settings, 264–266 overview of, 281
initial settings, 256–257 SSL and, 282
master daemon, 263–264 Cyrus IMAPd
overview of, 250–251 components, 275–276
simple mail server, 267–268 installing, 275
to handle inbound and outbound main configuration file, 276–277
managing user mailboxes, 277–278
mail, 251–255
to use lookup tables, 269–273
Samba servers
applying permissions to local direc-

tory, 232
defining share, 232–234
local directory to share, creating, 232

NI N D E X 387

D Distributed Replicated Block Device
(DRBD)
database
for Kerberos server, creating, 336 monitoring, 168–169
LDAP SAN and, 161
adding information to, 215–217 setting up, 164–165
deleting entries from, 222 starting, 166–167
modifying entries in, 221 dpkg-reconfigure postfix command,

Deadline scheduler (I/O scheduler), 97 257–262
debugfs utility dpkg-reconfigure slapd command,

Ext2/Ext3 file system, 122–124 203–206
overview of, 378 drbddisk resource type, 182
showing contents of inodes with, DRBL (Diskless Remote Boot in Linux),

110–111 setting up
debugreiserfs tool, 129–130 configuring software, 32–33
default page size, in memory, 57 DHCP server, 33–35
default schema files of LDAP, 200–201 installing software, 31–32
deleting entries from LDAP database, 222 overview of, 30
device-device option (Clonezilla), 39 drift file, NTP, configuring, 327
device-image option (Clonezilla), 39 dual-core server, monitoring performance
devices for LVM logical volumes, 374–375
DHCP server, setting up, 33–35 on, 47
Diffie-Hellman parameters, generating, dumpe2fs utility, 120–122

311–312 E
digital certificates, 282
directory e2fsck utility, 120
easy-rsa scripts, generating certificate
See also LDAP Directory
assigning for Clonezilla, 35–36 authority with, 308
description of, 111–112 editing Samba configuration file, 233
/etc/event.d, 347 e-mail, receiving
/etc/init.d, 347
Directory Components (LDAP), 198 Cyrus IMAPd, using, 275–278
disabling LDAP version 2 support, 206 overview of, 274
disk activity, and storage performance, 66 procmail, using, 278–279
disk layout Qpopper, using, 279–280
blueprint of, 6–7 e-mail, sending. See Postfix MTA
file system, choosing, 5–6 enterprise network installation
RAID, setting up, 4–5 64-bit version of, 7
Diskless Remote Boot in Linux (DRBL), authentication handling, 3
completing, 22–23
setting up connection to SAN, 2–3
configuring software, 32–33 file system, choosing, 5–7
DHCP server, 33–35 LVM logical volumes, creating, 16–22
installing software, 31–32 overview of, 1
overview of, 30 post-installation tasks
disk mirroring (RAID method), 4
disk striping (RAID method), 4 multipathing, setting up, 26–27
Distinguished Name (LDAP), 198 NIC bonding, setting up, 24–26
preparing for, 3–4
RAID, setting up, 4–5
server hardware, 2

388 NINDEX

software-based RAID, setting up, 9–16 description of, 6, 116
starting, 8 mounting, 119
entries in LDAP, 198 extents, description of, 112
using, 25
/etc/defaults/slapd file, 211–213 F
/etc/event.d directory, 347
/etc/ha.d/authkeys file, 176 fdisk command, 366–367
/etc/ha.d/ha.cf file, 175–179 fdisk -l command, 359
/etc/ietd.conf file, 169 Fibre Channel, 162
/etc/imapd.conf file, 276–277 files, backing up, snapshot technology for,
/etc/init.d directory, 347
/etc/init.d/networking script, 349–350 5
/etc/lvm/lvm.conf file, 373 file systems
/etc/mke2fs.conf configuration file
contents of, 116 accessing damaged Ext3, 378–380
options, 117–119 analyzing Ext2/Ext3, 119–124
/etc/modprobe.d/aliases, loading correct choosing at installation, 5–7
description of, 6, 109
kernal modules using, 25 indexing, 115
/etc/modprobe.d/arch/i386 file, contain- inodes and directories, 110–112
journaling, 114–115
ing correct bonding options mounting, 112–113, 119
/etc/nagios2/apache2.conf file, contents optimizing

of, 132 Ext2/Ext3, 116–119
/etc/nagios2/cgi.cfg file, contents of, 132 ReiserFS, 128–130
/etc/nagios2/commands.cfg file, contents XFS, 124–128
repairing
of, 135 Ext2/Ext3, 119–124
/etc/network/interfaces, creating bond() ReiserFS, 380
superblocks, inode bitmaps, and block
devices using, 26
/etc/pam.d/login file contents, 224 bitmaps, 112–113
/etc/postfix/main.cf file, parameters, 266 troubleshooting, 378
/etc/postfix/master.cf file filtering incoming e-mail, 278–279
fork() system call, 91
listing, 263 forwarding mail to servers on Internet, 267
predefined fields and default values, free -m command, 57
measuring performance, 86
264 fsck.reiserfs tool, 128–129
/etc/ssl/openssl.cnf file, 285
/etc/xinetd.d/qpopper configuration file, G

279 generic network performance optimiza-
ethtool utility, 74, 75 tion, 106–107
Ext2/Ext3 file system
getent command, 230
accessing damaged, 378–380 global settings for Postfix, configuring,
analyzing and repairing
264–266
debugfs utility, 122–124 gpart command, 365
dumpe2fs and tune2fs utilities, Grub

120–122 boot loader, 344
e2fsck utility, 120 configuring huge pages with, 92
creating, 116–119
command-line interface, 362

NI N D E X 389

loading, 362–364 inactive memory, 58
reinstalling, 361 inbound mail
troubleshooting, 361
Grub menu, 344, 351 from local user to local user, processing,
251–252
H
Postfix and, 251
ha.cf file, 175–179 sent over network to local user, process-
hardware initialization, 344
hardware interrupts, performance and, 52 ing, 253
hardware requirements for SAN, 163 indexing, description of, 115
hb_gui interface (Heartbeat), 180–183 init=/bin/bash tool, 351–353
Heartbeat initrd loading, 346
init script (Postfix), 262
configuring ha.cf file, 175–179 inode bitmaps, 112–113
hb_gui interface, 180–183 inodes, 110–112
SAN and, 162 installation
Heimdal Kerberos, 321
help command, 326 See also installing
help command (debugfs utility), 124 64-bit version of Ubuntu Server, 7
hierarchical structure of LDAP, 197 completing, 22–23
high-availability solution for Apache Web LVM logical volumes, creating, 16–22
on enterprise network
Server, 194
high memory, 59 authentication handling, 3
hi performance category (top utility), 49 connection to SAN, 2–3
Host Detail window (Nagios), 157 file system, choosing, 5–7
hosts and host groups for Nagios tool, overview of, 1
preparing for, 3–4
defining, 141–143 RAID, setting up, 4–5
huge pages, configuring, 92–93 server hardware, 2
post-installation tasks
I multipathing, setting up, 26–27
NIC bonding, setting up, 24–26
id performance category (top utility), 48 software-based RAID, setting up, 9–16
identifying problem for troubleshooting, starting, 8
installing
344–350 See also installation
ietadm command, 171 AppArmor, 293
ifconfig command, 73–74 Cyrus IMAPd, 275
imaging network, schematic overview of, Diskless Remote Boot in Linux software,
32
imaging server (Clonezilla) 31–32
Kerberos, 330–332
assigning directory for, 35 Nagios tool, 131
clients for cloning, configuring, 36–38 NRPE service on Linux servers, 152–153
cloning client, 39–43 OpenLDAP, 202
configuring, 36 OpenVPN, 303
Diskless Remote Boot in Linux Postfix, 256
software for SAN, 163
configuring software, 32–33 integrating Samba server
DHCP server, setting up, 33–35 in Active Directory
installing software, 31–32
setting up, 30
setting up, 29

390 NINDEX

Kerberos authentication, setting up, Kerberos
245 See also Kerberos server; NTP time
server
making Samba member of domain, authentication, setting up for Samba
244 servers, 245
client
with LDAP configuring, 339
configuring secure connections, 239 logging in with, 340–341
connecting Samba to LDAP, 238–239 description of, 321
preparing LDAP, 237–238 design goals for, 329
preparing Samba, 236–237 installing and configuring, 330–332
specifying where to put objects, 240 versions of, 321

Internet time, 323 Kerberos Distribution Center (KDC), 330,
inter-process communication, optimizing, 338–339

94–96 Kerberos server
interrupt counter, 53 configuring
interruptible hang, 375–377 database and stash file, 336
I/O scheduler generic settings, 332–335
KDC settings, 335–336
description of, 96 starting and creating administrative
optimizing, 97–98 user account, 337–338
iostat utility user accounts, adding, 338
disk performance and, 69 verifying KDC is operational, 338–339
-x option, 70, 71
ipcs -lm command, 94 kernel panic, 346
IPTraf tool kernel
Additional Ports option, 75
description of, 75 interruptible hang, 375–377
interface, 77 loading, 346
LAN station monitor, 77 noninterruptible hang, 378
IQN (iSCSI Qualified Name) of target, 169 parameters, tuning, 98–100
iscsiadm command, 173 symmetric multiprocessing, 88
iscsiadm -m session command, 174 troubleshooting, 375
iSCSI initiator, configuring, 173–175 kinit command, 338
iSCSI Qualified Name (IQN) of target, 169 klist command, 338
iSCSI target Knoppix Rescue CD, 357–360
configuring, 169–172 krb5.conf file
SAN and, 162 after installation, 332–335
iscsitarget resource type, 187 PAM settings, 340

J L

Journaled File System (JFS), 6 ldapadd command (LDAP), 201
journaling ldap.conf files, 229
ldapdelete command (LDAP), 201, 222
modes of, 114–115 LDAP (Lightweight Directory Access Pro-
ReiserFS and, 128
tocol)
K See also OpenLDAP
back-end database, 205
kadmin.local command, 337, 338 Data Interchange Format, 201
kdc.conf file listing, 335–336 default schema files, 200–201
KDC (Kerberos Distribution Center), 330,

338–339