Troubleshooting Slow Linux Systems

If you system is running slowly, and this goes for RHEL, Debian and other variants then take a look at this article which is a simple walkthrough of the tools you can use to solve problems.  These specific examples are from a system running Openstack, but that’s not important to most of you:

  • top – The place to start is generally the ‘top’ command which shows a resource summary and task list.
  • iotstat – Shows the reads and writes on your disk
  • iotop – Realtime iostat
  • iozone – Generate some test traffic to see how the system reacts.

Continue reading

SSL Certificate – Renewing a Certificate for ApacheSSL

This doc was written to assist with the next time I have to update the SSL cert on an ApacheSSL server. The specific examples here are from an RHEL5 server with Apache2.x

Here are the steps.

  1. Locate the Relevant Files - These files are the CRT and CSR. The CRT is issued by Thawte or Verisign, and the CSR is the request that you send to them. To find these locate your httpd.conf file or possibly the httpd-ssl.conf file in ./extras/. This stuff should typically be in /etc/httpd/conf, but idiots will place it in other locations because they don’t understand the UNIX conventions. Specifically you are looking for a line that says something line this. SSLCertificateKeyFile /usr/local/apache2/conf/server.key. That will take you to server.key as the file. You need to find the CSR associated with the key and copy that.
  2. Login to the Certificate Authority’s Web Site - For Thawte, we have an account itsupport/Thawt3SSL and fill out the information. If you get stuck, the reference the current ‘live’ certificate with your browser. You will need to paste the… sadly my notes just trail off here.
  3. Certificate Authority will Call You - So you need to be able to answer, or have someone fwd the Cert. Authority to your phone. The just want to verify things.
  4. Apply the Certificate - This will need to go in the file that is listed in your config files… see #1 above, but, SSLCertificateFile /usr/local/apache2/conf/server.crt it’s the .crt file not the key.
  5. Restart Apache - service httpd restart
  6. Check the Cert - for the new expiration date.

NB: if you need to make a change to any information, then the certificate authority will need to talk with company executives, have some faxes sent and generally draw the whole thing out. Also you will need to generate a new key. You will need to start with key generation:

Code:

/usr/bin/openssl genrsa -rand /dev/urandom -out /usr/local/apache2/conf/server_new.key 1024
/usr/bin/openssl req -new -key /usr/local/apache2/conf/server.key -out /usr/local/apache2/conf/server.csr

Create the .key and then the request .csr. This is where you make the changes to the information and the CSR is what you will submit to the authority.

PostgreSQL Replication to a Warm-Standby Using WAL Files

THEORY

Like a good relational database, PostgreSQL maintains a set of transactional log files known as write-ahead-logs (WAL) in the pg_xlog directory.  These logs are written to for every change in the database files and are used to recovery from a crash condition.  If you crash, replay all the WAL files since the last backup and you will be back in business right at the point of failure.

Well, if you have this capability, what about keeping a warm-standby system and feeding it all the WAL files.  If you teach the warm-standby how to continuously process the incoming write ahead logs from the live system you will have a system ready to go at a moments notice.  When you read about this setup in other places online the primary server is known as ‘master’ and the secondary the ‘slave’.

NOTE BENE: Both your primary and your secondary need to be running the same major version of the postgreSQL database. Continue reading

Restoring Files From RackSpace Cloud Files

If you are like me and have a cloud server on rackspace you probably have a backup of your server that runs weekly or daily but may have never found a nice way to access these files.  In fact, i was on chat with a Fanatical Support guy the other day shortly after I had deleted my httpd.conf file.  I asked him if I could restore a file using my cloud file backups and he said “No”.

That bothered me, but I don’t expect support the guys to be all knowing, even if it is a top-notch organization like Rackspace.  The real answer is yes.  Here is how it’s done.

If you are familiar with the API calls for interacting with RackSpace programmatically, you should probably skip this article, it’s going to be really basic.  If you want to learn these calls, then I found a nice article here that describes pulling and extracting the files for a Windows image and getting a .vhd file

ANATOMY OF A BACKUP

So logging in to the RackSpace Cloud interface and you should see a new(ish) addition to the Hosting Menu.  Choose “Cloud Servers” under the Open Cloud and then you’ll enter a new interface.  Once there click on “Files”  At this point you see your files.  Yes, you can see them in the old interface, but you cannot download them.

What I found was a set of files with a timestamp in them and a site ID.  One meta file that ends and .yml and describes all of the other compressed tarballs that contain the actual data.  You probably noticed that the tarballs are incremented (0, 1, 2, etc)

---
name: daily_20120827_111111_cloudserver1111111.yml
 format: tarball
 image_type: full
 files:
 - daily_20120827_111111_cloudserver111111.tar.gz.0
 - daily_20120827_111111_cloudserver111111.tar.gz.1
 - daily_20120827_111111_cloudserver111111.tar.gz.2

WHAT TO DO WITH THEM

If you have all the files in one directory you should be able to address them line this.  Remember, I’m trying to find my httpd.conf.  Well, this is going to find any and all httpd.conf file in the tar.gz files available.

for tarball in `ls -1 *cloudserver111111.tar.gz.*`
do
    recoveryfile=`tar -tzf $tarball | grep httpd.conf`
    tar -zxvf $tarball $recoveryfile
done

You will want to change the file you are looking for (httpd.conf) and the first line which defines the files you want to look through.  I’d use the find * command at the end to expose the directory structure that was created.

Viewing Your Linux Hardware with DMIDECODE

I never like opening a running system when I can simply query that system with a simply command for the information needed.  dmidecode is a great tool for polling hardware information in human-readable format.

In its simplest form you will dump all the information to the screen

dmidecode

but that’s a bit much so try running with the -t argument which lets you narrow down the search to the components (bios, system, baseboard, chassis, processor, memory, cache, connector, slot)  So, for instance, if need to learn how much RAM you system can handle:

# dmidecode -t memory
# dmidecode 2.10
SMBIOS 2.7 present.
# SMBIOS implementations newer than version 2.6 are not
# fully supported by this version of dmidecode.

Handle 0x0027, DMI type 16, 23 bytes
Physical Memory Array
    Location: System Board Or Motherboard
    Use: System Memory
    Error Correction Type: Single-bit ECC
    Maximum Capacity: 32 GB
    Error Information Handle: No Error
    Number Of Devices: 4

Enjoy and let me know you you end up using this command.

 

REFERENCE – Files edited for Pentaho Setup.

These are the files I edited when setting up Pentaho on a RedHat/CentOS server.  This is really for personal reference, but if you have any questions I may be able to help

/opt/pentaho/biserver-ce/pentaho-solutions/system/applicationContext-spring-security-jdbc.xml
/opt/pentaho/biserver-ce/pentaho-solutions/system/dialects/mysql5/hibernate-settings.xml
/opt/pentaho/biserver-ce/tomcat/webapps/pentaho/META-INF/context.xml
/opt/pentaho/biserver-ce/pentaho-solutions/system/publisher_config.xml
/opt/pentaho/biserver-ce/tomcat/webapps/pentaho/WEB-INF/web.xml
/opt/pentaho/biserver-ce/tomcat/conf/server.xml
/opt/pentaho/biserver-ce/pentaho-solutions/system/pentaho.xml
/opt/pentaho/biserver-ce/pentaho-solutions/system/pentaho-spring-beans.xml
/opt/pentaho/biserver-ce/pentaho-solutions/system/applicationContext-spring-security-ldap.xml
/opt/pentaho/biserver-ce/pentaho-solutions/system/smtp-email/email_config.xml

# Authentication Location
/opt/pentaho/biserver-ce/pentaho-solutions/system/applicationContext-security-ldap.properties
/opt/pentaho/biserver-ce/pentaho-solutions/system/applicationContext-spring-security-ldap.xml

Installing Eclipse & VMWare Studio Explorer on Mac OSX Lion

Older version of the Mac OS require you to make a visit to Apple’s development site and download a JDK, but if you have Lion then you simply need to choose the proper download from the Eclipse Site.  I used Eclipse Classic 3.7.2.

  1. Visit the Eclipse Download Site and grab Eclipse Classic.
  2. It comes as a tar.gz file.  Expand it an place it in you Applications directory.
  3. Double-click on the Eclipse Application.
  4. “Preferences | Install/Update | Available Software Sites” and choose “Add”
  5. Enter the location of your http://10.1.1.101/eclipse/update and give it a name like “VMWare VM Studio”
  6. Click the “Reload” button.  If your new site is properly setup it should come back error free and you can close that dialog.
  7. Click “Help | Install New Software…” and use the drop down to select your new site.
  8. Use the triangle to explore the available software, choose “VMWare Studio Plugin for Eclipse” and select the “Next” button.
  9. After working out all the dependencies, the installer will require you to accept a licensing agreement.  Click “Finish” when done.
  10. Select “Window | Show View | Other” and locate your VMware Studio Explorer.  Select it and click okay.
  11. You will have to restart Eclipse.

Once it comes back you will have a new explorer tab and a fine place to administer the VMware Studio from.

 

 

Tuning mySQL – Because by default it’s not even close to tuned.

Basic tuning of the mySQL is accomplished in the /etc/my.cnf file. If you want to get all geeky and into this reference the seminal document over on the mysql dev site. This should result in a speed increase in your system.  It certainly has in my system running mySQL 5.x.

The information below is expressed as a set of ratios that begins with your system RAM and then works from there.

innodb_buffer_pool_size = $SYSTEMRAM/2
innodb_additional_mem_pool_size = $innodb_buffer_pool_size/20
innodb_log_file_size = $innodb_buffer_pool_size/4
innodb_log_buffer_size = $innodb_buffer_pool_size/50 or a minimum value of 8MB

Note bene: Changing your log file size can results in a mySQL refusing to start.  Simply remove these files from you mysql data directory and they will be created on the next startup.

Script to Move Database Location – mySQL

Don’t run this script.  It’s a concept that I haven’t tested and running it is pretty well guaranteed to crash your mysql server.  It’s designed to make the relocation of data faster, but I don’t have time to finish it today.

You should probably use this fellow link because it works… it’s just slower and manual.  Oh, and if you do get a scripting urge, please make this script work properly for me and post it in a comment.  Thanks.

 

USER=root
PASSWORD=yourpassword
DBS="$(mysql --user=$USER --password=$PASSWORD -Bse 'show databases')"
OLDDATA_DIR="/var/lib/mysql"
NEWDATA_DIR="/database/lib/mysql"

mkdir -pv $NEWDATA_DIR

for FILE in ${DBS[@]}; do
        DATABASE=`basename $FILE`
        echo cp -R $OLDDATA_DIR/$DATABASE $NEWDATA_DIR/$DATABASE
done

# Set permissions
chown -R mysql:mysql $NEWDATA_DIR

# Archive the old & link it to the new
mv $OLDDATA_DIR OLDDATA_DIR-old
ln -s $NEWDATA_DIR/$DATABASE $OLDDATA_DIR/$DATABASE

#get_mysql_option mysqld datadir "/database/lib/mysql"
sed -i  's|$OLDDATA_DIR|$NEWDATA_DIR|' /etc/init.d/mysqld
sed -i  's|$OLDDATA_DIR|$NEWDATA_DIR|' /etc/my.cnf

Setting up Apache Log File Rotation

This how-to walks users through setting up proper log file rotation for a multil-site Apache installation where the log file are broken out by site. I built all this on my own, but forgot about logfile rotation so now the log files just keep growing and growing.  Time to institute a log rotation algorithm.

For the most part when you are working with Unix you will find that the syslog daemon handles how messages are logged in you system, but Apache handles it’s own logs and the details are typically kept in the httpd.conf file.

sudo grep -i 'log' /etc/httpd/conf/httpd.conf /etc/httpd/conf.d/*
# Custom log file locations
LogLevel warn
ErrorLog  /var/www/html/site1.com/log/error.log
CustomLog /var/www/html/site1.com/log/access.log combined
# Custom log file locations
LogLevel warn
ErrorLog  /var/www/html/site2.com/log/error.log
CustomLog /var/www/html/site2.com/log/access.log combined
# Custom log file locations
LogLevel warn
ErrorLog  /var/www/html/site3.com/log/error.log
CustomLog /var/www/html/site3.com/log/access.log combined

So, grepping gives me a listing of logfile locations for each of the sites and as you can see they are all located in different directories.  You probably also noticed that there are logfiles in the con.d directory that I grepped for.  A lot of stuff will want to install there, like phpMyAdmin or webalizer or ssl.conf.  One other note, some installations will have their config files in an apache2 directory. Continue reading