Troubleshooting Slow Linux Systems

If you system is running slowly, and this goes for RHEL, Debian and other variants then take a look at this article which is a simple walkthrough of the tools you can use to solve problems.  These specific examples are from a system running Openstack, but that’s not important to most of you:

  • top – The place to start is generally the ‘top’ command which shows a resource summary and task list.
  • iotstat – Shows the reads and writes on your disk
  • iotop – Realtime iostat
  • iozone – Generate some test traffic to see how the system reacts.

Continue reading

SSL Certificate – Renewing a Certificate for ApacheSSL

This doc was written to assist with the next time I have to update the SSL cert on an ApacheSSL server. The specific examples here are from an RHEL5 server with Apache2.x

Here are the steps.

  1. Locate the Relevant Files - These files are the CRT and CSR. The CRT is issued by Thawte or Verisign, and the CSR is the request that you send to them. To find these locate your httpd.conf file or possibly the httpd-ssl.conf file in ./extras/. This stuff should typically be in /etc/httpd/conf, but idiots will place it in other locations because they don’t understand the UNIX conventions. Specifically you are looking for a line that says something line this. SSLCertificateKeyFile /usr/local/apache2/conf/server.key. That will take you to server.key as the file. You need to find the CSR associated with the key and copy that.
  2. Login to the Certificate Authority’s Web Site - For Thawte, we have an account itsupport/Thawt3SSL and fill out the information. If you get stuck, the reference the current ‘live’ certificate with your browser. You will need to paste the… sadly my notes just trail off here.
  3. Certificate Authority will Call You - So you need to be able to answer, or have someone fwd the Cert. Authority to your phone. The just want to verify things.
  4. Apply the Certificate - This will need to go in the file that is listed in your config files… see #1 above, but, SSLCertificateFile /usr/local/apache2/conf/server.crt it’s the .crt file not the key.
  5. Restart Apache - service httpd restart
  6. Check the Cert - for the new expiration date.

NB: if you need to make a change to any information, then the certificate authority will need to talk with company executives, have some faxes sent and generally draw the whole thing out. Also you will need to generate a new key. You will need to start with key generation:

Code:

/usr/bin/openssl genrsa -rand /dev/urandom -out /usr/local/apache2/conf/server_new.key 1024
/usr/bin/openssl req -new -key /usr/local/apache2/conf/server.key -out /usr/local/apache2/conf/server.csr

Create the .key and then the request .csr. This is where you make the changes to the information and the CSR is what you will submit to the authority.

Viewing Your Linux Hardware with DMIDECODE

I never like opening a running system when I can simply query that system with a simply command for the information needed.  dmidecode is a great tool for polling hardware information in human-readable format.

In its simplest form you will dump all the information to the screen

dmidecode

but that’s a bit much so try running with the -t argument which lets you narrow down the search to the components (bios, system, baseboard, chassis, processor, memory, cache, connector, slot)  So, for instance, if need to learn how much RAM you system can handle:

# dmidecode -t memory
# dmidecode 2.10
SMBIOS 2.7 present.
# SMBIOS implementations newer than version 2.6 are not
# fully supported by this version of dmidecode.

Handle 0x0027, DMI type 16, 23 bytes
Physical Memory Array
    Location: System Board Or Motherboard
    Use: System Memory
    Error Correction Type: Single-bit ECC
    Maximum Capacity: 32 GB
    Error Information Handle: No Error
    Number Of Devices: 4

Enjoy and let me know you you end up using this command.

 

Mounting a ‘Foreign’ LVM Volume

First, what do I mean by foreign?  Foreign means mounting the logical volume with an OS that it wasn’t originally installed on.  This could be because you are using KNOPPIX to repair something on the volume, or because you’ve moved the disk to a new location.

The process itself if quite simple, but it would help if you understood how logical volumes work first.  Click here for some nice background

Mounting an LVM Volume

  1. First Identify it with the fdisk command
  2. And find the VolGroup with the pvs command
  3. lvdisplay will show you the Logical Volume
  4. Finally mount it

# fdisk -l
Disk /dev/sda: 500.1 GB, 500107862016 bytes255 heads, 63 sectors/track, 60801 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes
Device Boot      Start         End      Blocks   Id  System/dev/sda1   *           1          13      104391   83 Linux/dev/sda2              14       60801   488279610   8e  Linux LVM

# pvs  PV         VG         Fmt  Attr PSize   PFree
/dev/sda2  VolGroup00 lvm2 a-   465.66G    0

lvdisplay

Okay, this isn’t finished yet, but I published it so that next time I’m working on this task I’ll complete it.  If you have any suggestions or want to complete this list. let me know.

About Jay Farschman - Jay currently works as a Senior Systems Administrator for an asset management company in Colorado where he works with companies that produce hardware, telecommunications software and financial services.  Jay previously owned a consulting company and provided training and consulting services for three Fortune 500 companies and numerous small businesses where he leveraged Linux to provided exceptional value.

Installing a System with Spacewalk

I am not going to cover the installation of Spacewalk in this article as there are nice articles on the spacewalk site detailing most of what you need to install it properly.  Yes, it’s tricky and yes, if you have any install questions I’ll do my best to help with those.  However, the goal of this document is to talk about using Spacewalk after you have it installed and configured it.  So this is more of and end user’s guide.  The links directly below are how I got were I was going.

# Installing Spacewalk – The installation
https://fedorahosted.org/spacewalk/wiki/HowToInstall
http://wiki.centos.org/HowTos/PackageManagement/Spacewalk

# Setting up DHCP in Windows
http://unattended.sourceforge.net/pxe-win2k.html

# Setting up Spacewalk – Post install
https://fedorahosted.org/spacewalk/wiki/HowToKickstartCobbler 

IN A NUTSHELL

There are  three ways to force a system rebuild.

  1. Interacting with the PXE menu – Yes, actually touching the system.
  2. Using the System Record in Cobbler
  3. Using Koan from the system to rebuild the system.

Terminology

Cobbler – Uses DHCP, TFTP and DNS to enable network-based installs.  Spacewalk does a lot of writing to cobbler to get things done, but there is a handy command line too.

Koan – A traditional koan is a story told by a Zen master to help enlighten a student.  Little things like “What is the sound of one hand clapping, grasshopper?”  In our case koan is an RPM package that is installed to enable us to request a system rebuild from within the system.

PXE – is the Preboot eXecution Environment that loads on any decent server prior to the OS.  With PXE we can boot from the network and give cobbler and spacewalk a chance at guiding the installation

Spacewalker – Spacewalk is an open source system management system that is upstream from the Redhat Satellite Server.  This means that you and I can run the same software distribution system they use at Redhat which is nice.  For one thing you get a web interface, and a robust system that was built for work in vary large organizations.

HOW TO DO IT

Assuming you have Spacewalk all setup and working this is all pretty simple, but I’ll be editing this article a bit until I’ve worked out all the issues.  Please help me to learn for your experiences as well.

From the PXE Menu – On my Dell servers generally hitting the F12 key on boot will invoke the PXE menu.  Once you invoke the PXE menu and having done nothing else you will see a menu of potential profiles.  Select one and you system will rebuild with that kickstart.

Using Cobbler – This can be done remotely, but you’ll need to ……

sorry, I have to work on other things at the moment.  No more documentation time.

cobbler system add –name <nameOfYourSystem> –mac <mac addr of netboot interface> –profile <a profile from ‘cobbler profile list’>

If you have already added the system to cobbler and simply want to have it rebuild on the next boot use this command:

cobbler system edit –name <nameOfYourSystem> –netboot-enabled=1

koan –replace-self –server=cobbler.example.org [--profile=profile-name] [--system=system-name]

About Jay Farschman - Jay currently works as a Senior Systems Administrator for an asset management company in Colorado where he works with companies that produce hardware, telecommunications software and financial services.  Jay previously owned a consulting company and provided training and consulting services for three Fortune 500 companies and numerous small businesses where he leveraged Linux to provided exceptional value.

TCPDUMP – Fast and Easy

TCPDump is a fine way to find out what a system is doing with another system, but generally you will see so much noise unless you limit the traffic that it will be difficult to see what’s happening.  This command did the trick for me when analyzing a problem with FTP

tcpdump src net 10.1.1.73 or dst net 10.1.1.73 -e -vv -w FTP_from_73.cap

Sometimes you may just want to look at dhcp information.

tcpdump -lenx -i eth0 -s 1500 port bootps or port bootpc

I ran this on the FTP server to capture the bare minimum of frames.  We wanted to get both sides of the conversation, the source (src) and the destination (dst) so that we have all the communication between the two systems.  One thing that you may be missing in this scenario is when one or both of these system call out to a third server, like a DNS server.  If you need to do that, just tcpdump everything.

So how do you view this.  Well, I creates a .cap file that is visible and filterable in wireshark which you can load with yum or apt-get.

About Jay Farschman - Jay currently works as a Senior Systems Administrator for an asset management company in Colorado where he works with companies that produce hardware, telecommunications software and financial services.  Jay previously owned a consulting company and provided training and consulting services for three Fortune 500 companies and numerous small businesses where he leveraged Linux to provided exceptional value.

Nmap with three commands

Network Mapper is an essential tool for every SysAdmin.  You need to probe around the network to make sure people haven’t left ports open in a haphazard manner.   These commands should get you ther.

Ping sweep and a reverse DNS shows you who is on the logical network with you:

nmap -sP 10.1.1.0/24

-sS does a bit of scanning for ports

nmap -sS 10.1.1.23

Or you can set off all the alarms on the network with the following pots scan of everything.

nmap -O 10.1.1.0/24

About Jay Farschman - Jay currently works as a Senior Systems Administrator for an asset management company in Colorado where he works with companies that produce hardware, telecommunications software and financial services.  Jay previously owned a consulting company and provided training and consulting services for three Fortune 500 companies and numerous small businesses where he leveraged Linux to provided exceptional value.

 

Relaying from RackSpace

Okay, I made an awful lot out of this because I was trying to use sendmail as my MTA.  Simply put that was stupid.  Sure you can do it with sendmail, but it’s about 10 times easier to use postfix.  We are going to use SMTPAUTH just to make all the security guys happy.

chkconfig sendmail off
yum remove sendmail
yum install postfix cyrus-sasl-plain cyrus-sasl-md5

You are almost done.  Edit /etc/postfix/main.cf and add these lines.  Nope, doesn’t matter where:

relayhost = mail.relay.com
smtp_sasl_auth_enable=yes
smtp_sasl_password_maps=hash:/etc/postfix/sasl_passwd
smtp_sasl_mechanism_filter = AUTH LOGIN
smtp_sasl_security_options =

You probably noticed the sasl_passwd file.  That needs a the instructions for setting up the SMTPAUTH so do that in /etc/postfix/sasl_passwd and be sure to set it so that root has rw and the group/everyone permissions are off (chmod 700 /etc/postfix/sasl_passwd):

mail.relay.com smtpauthuser@relay.com:smtpauthpassword
postmap  /etc/postfix/sasl_passwd
postmap /etc/postfix/generic
chkconfig postfix on
service postfix start

Q.E.D

About Jay Farschman - Jay currently works as a Senior Systems Administrator for an asset management company in Colorado where he works with companies that produce hardware, telecommunications software and financial services.  Jay previously owned a consulting company and provided training and consulting services for three Fortune 500 companies and numerous small businesses where he leveraged Linux to provided exceptional value.

Using fPing to look at a Range

The fping utility is a f-ing useful with a simple command line.  One word of caution.  If you work with other network administrators they sometimes don’t take kindly to having their network ping scanned.  In fact, there are a number of products out that that will detect, log and and some case take actions against the originating IP Address.  Play nice.

$ fping -c1 -gds 10.1.1.0/24 2>&1| egrep -v "ICMP|xmt" 

10.1.1.1        : [0], 96 bytes, 0.23 ms (0.23 avg, 0% loss)

Play around with it, but it going to scan the range and display something like the above entry.

About Jay Farschman - Jay currently works as a Senior Systems Administrator for an asset management company in Colorado where he works with companies that produce hardware, telecommunications software and financial services.  Jay previously owned a consulting company and provided training and consulting services for three Fortune 500 companies and numerous small businesses where he leveraged Linux to provided exceptional value.

Building Amanda on CentOS or Redhat.

This is a modern interpretation of some work I did some time ago with version 2.6.1.  I’m just now working with 3.3.1 and documenting as I work so you may find that it’s not complete today if I get interrupted.

Jay – 27 July 2011.

IN A NUTSHELL

  1. Install the OS
  2. Patch the OS
  3. Load AMANDA Server Software
  4. Configure AMANDA
  5. Verify the Install

PREREQUISITES
We will need a system with gigabit interfaces, a large (1TB) drive space to act as virtual tapes, USB 2.0 capable ports and at least 2GB of RAM.  While the backup server does not strictly need to be a high powered server it doesn’t hurt to give it some guts as the remote servers are often not powerful enough to do the compression/encryption of the data before transmission.  A powerful backup server can easily handle task of building tar.gz files.

STEP-BY-STEP
These are the steps mentioned above and while you may want to vary from them I would not recommend skipping steps like “Patch the OS” as this is the sort of thing that can come back and bit you. Patching can take some time, but don’t skip this step. For the most part patching is an unattended step so you can do other things.

### Install the OS ###
I would recommend using a CentOS install for this as there are RPMs available which will facilitate later steps. When building try to keep the install minimal You don’t need that much to make AMANDA work, however, you should setup a few things:

  • Operate on the CLI (init 3) rather then with a GUI.  Just change the /etc/inittab so that the first uncommented line is id:3:initdefault: instead of id:5:initdefault:
  • Firewall with 10080-10083 tcp/udp open. – Let me know if you need firewall settings.
  • NO SELINUX – Unless you want to help me define these?

### Patch the OS ###
Do not skip this step. Even though it may take an hour to load the updates it it well worth the time. Running on unpatched code is just begging for a weird, time consuming problem later in this process. While this is happening you can work on the path statement below and also cleaning up the start up files.

# Load Updates
 yum -y upgrade
 reboot
 

While you are waiting for yum to finish you can work on a few other things as well.

# Path Changes
for d in /usr/local/bin /usr/local/sbin
do
 case :$PATH: in
 *:$d:*) : ;;
 *) PATH=$d:$PATH ;;
 esac
done
# Turn off extraneous processes (NB: there are plenty of new processes in RHEL6)
chkconfig autofs off
chkconfig cups off
chkconfig ip6tables off
chkconfig bluetooth off

### Load AMANDA Server Software ###
Grab the latest Amanda software from the RPM repository here and install it. Don’t be too worried about the fact that the version don’t match. Assuming the config file doesn’t change the location of the binaries… which we double-check later… there should not be a problem. Your only issue is you may be missing a feature that makes your AMANDA experience “better”:

http://www.zmanda.com/download-amanda.php

Something like this:

mkdir -pv ~/addon_software; cd ~/addon_software
wget http://www.zmanda.com/downloads/community/Amanda/3.3.0/Redhat_Enterprise_4.0/amanda-backup_server-3.3.0-1.rhel4.x86_64.rpm
wget http://www.zmanda.com/downloads/community/Amanda/3.3.0/Redhat_Enterprise_5.0/amanda-backup_client-3.3.0-1.rhel5.x86_64.rpm
wget http://www.zmanda.com/downloads/community/Amanda/3.3.0/Redhat_Enterprise_4.0/amanda-backup_client-3.3.0-1.rhel4.i386.rpm
 yum localinstall --nogpgcheck amanda-backup_server*.rpm

 # Check for errors
 cat /var/log/amanda/install.err
# Set the amandabackup user password and unlock the account
 passwd amandabackup
 passwd -u amandabackup

# You may also want to get ntpd setup and running - not discussed here
# but having accurate time will most certainly help things along.

—–

### Amanda Admin Information
Because any newer version of the Amanda Server RPM could make changes to the usernames, groups and default directories you may want to check these with the amadmin command. This command will display what your current installation has set for users, groups, and directories. If there is a difference when you recover the amanda configuration files from the backups you will need to make the appropriate changes. I have included the expected responses in the comment lines:

# Amanda User - amandabackup
 /usr/sbin/amadmin xx version | grep CLIENT_LOGIN
# Amanda Configuration Directory - /etc/amanda
 /usr/sbin/amadmin xx version | grep CONFIG_DIR
# Amanda Debug Log Dir - /tmp/amanda
 /usr/sbin/amadmin xx version | grep AMANDA_DBGDIR
# Amanda Executables - /var/lib/amanda
 /usr/sbin/amadmin xx version | grep libexecdir
# Amanda GNUTAR Lists - /var/lib/amanda/gnutar-lists
 /usr/sbin/amadmin xx version | grep listed_incr_dir

# Create the Holding Disk
You may want place your holding disk on a high-speed drive that is separated from the OS drive. This will speed things up a bit.

# Setup a Holding Disk.  Why? Well if you have tapes that are slower
 # than your hard drive a holding disk will grab the data faster. Also,
 # holdingdisks will continue to grab incremental backups if you fail to
 # change the tapes.  Kind of a cool thing that.
 #
 # For this specific install I have a HUGE 12TB array that I can use.
 # Gadzooks!  I called it /storage 
mkdir -pv /storage/holdingdisk
 mkdir -pv /storage/vtape
 chown -R amandabackup:disk /storage/holdingdisk /storage/vtape
# Setup the amanda permissions in the "secret file"
 # it's really not a secret, but lots of people forget this step.  If
 # you add in all the servers and give them permissions to interact with
 # amanda 
vi /var/lib/amanda/.amandahosts
 
servername.domain.com    root amandabackup amandad amindexd amidxtaped
servername.domain.com    amandad amidxtaped
# This gives two different sets of permission.  One for the root user
 # and the other for the amanda daemon.
# Setup the /etc/amanda/amanda-client.conf with a valid tape device
 # in my case tapedev         "file:/storage/vtape"   # your tape device

# Build your tapes
Note that your backups will appear in one or more of the slots which represent both the full and the incremental backups located inside /storage/vtape. You’ll want to create all the tapes.  I believe there is a nice clean way to create these labels. but you can also us this lovely DO LOOP:

cd /storage/vtape
for ((i=1; $i<=15; i++)); do mkdir slot$i; done
for ((i=1; $i<=15; i++)); do /usr/sbin/amlabel daily daily-$i slot $i; done

I had some difficulty here where even though I labeled my tapes I was getting messages about ‘unlabeled volume’ on all my tapes.  So I ended up setting the permissions so that amandabackup:disk owned the vtapes/slots and then ran things as amandabackup  Suddenly the results of my “amtape daily show” where no longer “unlabeled volume” but label daily-nn.

# Reset the tapes
amtape daily reset

#### Configuration #####
Skipped for now, please contact me and i’ll share my config and any of the intricacies that I have found over the years.

Setup Crontab

# Amanda check and dump
0 16 * * 1-7 /usr/sbin/amcheck -m daily
05 21 * * 1-7 /usr/sbin/amdump daily
0 8 * * * find /etc/amanda/daily/log.20* -type f -mtime +25 -exec rm {} \;

.
#### Test Amanda #####
Login as amandabackup and run amcheck daily.  This will generate some errors with specific messages about changes you need to make to the .amandahost files of your clients.

### Other Tips ####

yum localinstall –nogpgcheck pigz-2.1.6-1.el5.rf.x86_64.rpm

amservice engineer.serverdomain.com bsdtcp noop < /dev/null

 

m