Setting up Simple MySQL Database Replication

ABOUT THE SETUP
I operate a busy OpenStack environment that is used by lots of people making changes everyday.  A typical day will see over 100 images spawned and terminates, but some of these last longer than a day.  Because OpenStack maintains a stateDB mapping the OpenStack layers to the KVM/Libvirt layers it’s critical to have not only backups, but mysql replication in place.  A failure 18 hours after my last backup could leave me scrambling through over 100 instances looking and performing some horrible manual fixes.

With that in mind I’m building a second controller for OpenStack which will keep a replicated copy of the database.  The plan is to add an HAProxy in to the mix to manage all connections from the Compute Nodes to the Controllers as well as incoming https connections from my users into the Controllers.

In the end, that’s where I’m going.  This article is just about the first part.  MySQL Replication.  BTW: This is Ubuntu 12.04

REFERENCE: https://dev.mysql.com/doc/refman/5.0/en/replication-howto.html

Like all of the documentation from MySQL it’s written at a pretty high-level and is 100% factually correct while also being slightly unclear and missing steps.   What is wrong with examples?  Why not add some?

Continue reading

Troubleshooting Slow Linux Systems

If you system is running slowly, and this goes for RHEL, Debian and other variants then take a look at this article which is a simple walkthrough of the tools you can use to solve problems.  These specific examples are from a system running Openstack, but that’s not important to most of you:

  • top – The place to start is generally the ‘top’ command which shows a resource summary and task list.
  • iotstat – Shows the reads and writes on your disk
  • iotop – Realtime iostat
  • iozone – Generate some test traffic to see how the system reacts.

Continue reading

Restoring Files From RackSpace Cloud Files

If you are like me and have a cloud server on rackspace you probably have a backup of your server that runs weekly or daily but may have never found a nice way to access these files.  In fact, i was on chat with a Fanatical Support guy the other day shortly after I had deleted my httpd.conf file.  I asked him if I could restore a file using my cloud file backups and he said “No”.

That bothered me, but I don’t expect support the guys to be all knowing, even if it is a top-notch organization like Rackspace.  The real answer is yes.  Here is how it’s done.

If you are familiar with the API calls for interacting with RackSpace programmatically, you should probably skip this article, it’s going to be really basic.  If you want to learn these calls, then I found a nice article here that describes pulling and extracting the files for a Windows image and getting a .vhd file

ANATOMY OF A BACKUP

So logging in to the RackSpace Cloud interface and you should see a new(ish) addition to the Hosting Menu.  Choose “Cloud Servers” under the Open Cloud and then you’ll enter a new interface.  Once there click on “Files”  At this point you see your files.  Yes, you can see them in the old interface, but you cannot download them.

What I found was a set of files with a timestamp in them and a site ID.  One meta file that ends and .yml and describes all of the other compressed tarballs that contain the actual data.  You probably noticed that the tarballs are incremented (0, 1, 2, etc)

---
name: daily_20120827_111111_cloudserver1111111.yml
 format: tarball
 image_type: full
 files:
 - daily_20120827_111111_cloudserver111111.tar.gz.0
 - daily_20120827_111111_cloudserver111111.tar.gz.1
 - daily_20120827_111111_cloudserver111111.tar.gz.2

WHAT TO DO WITH THEM

If you have all the files in one directory you should be able to address them line this.  Remember, I’m trying to find my httpd.conf.  Well, this is going to find any and all httpd.conf file in the tar.gz files available.

for tarball in `ls -1 *cloudserver111111.tar.gz.*`
do
    recoveryfile=`tar -tzf $tarball | grep httpd.conf`
    tar -zxvf $tarball $recoveryfile
done

You will want to change the file you are looking for (httpd.conf) and the first line which defines the files you want to look through.  I’d use the find * command at the end to expose the directory structure that was created.