Setting up Simple MySQL Database Replication

ABOUT THE SETUP
I operate a busy OpenStack environment that is used by lots of people making changes everyday.  A typical day will see over 100 images spawned and terminates, but some of these last longer than a day.  Because OpenStack maintains a stateDB mapping the OpenStack layers to the KVM/Libvirt layers it’s critical to have not only backups, but mysql replication in place.  A failure 18 hours after my last backup could leave me scrambling through over 100 instances looking and performing some horrible manual fixes.

With that in mind I’m building a second controller for OpenStack which will keep a replicated copy of the database.  The plan is to add an HAProxy in to the mix to manage all connections from the Compute Nodes to the Controllers as well as incoming https connections from my users into the Controllers.

In the end, that’s where I’m going.  This article is just about the first part.  MySQL Replication.  BTW: This is Ubuntu 12.04

REFERENCE: https://dev.mysql.com/doc/refman/5.0/en/replication-howto.html

Like all of the documentation from MySQL it’s written at a pretty high-level and is 100% factually correct while also being slightly unclear and missing steps.   What is wrong with examples?  Why not add some?

Continue reading

Restoring Files From RackSpace Cloud Files

If you are like me and have a cloud server on rackspace you probably have a backup of your server that runs weekly or daily but may have never found a nice way to access these files.  In fact, i was on chat with a Fanatical Support guy the other day shortly after I had deleted my httpd.conf file.  I asked him if I could restore a file using my cloud file backups and he said “No”.

That bothered me, but I don’t expect support the guys to be all knowing, even if it is a top-notch organization like Rackspace.  The real answer is yes.  Here is how it’s done.

If you are familiar with the API calls for interacting with RackSpace programmatically, you should probably skip this article, it’s going to be really basic.  If you want to learn these calls, then I found a nice article here that describes pulling and extracting the files for a Windows image and getting a .vhd file

ANATOMY OF A BACKUP

So logging in to the RackSpace Cloud interface and you should see a new(ish) addition to the Hosting Menu.  Choose “Cloud Servers” under the Open Cloud and then you’ll enter a new interface.  Once there click on “Files”  At this point you see your files.  Yes, you can see them in the old interface, but you cannot download them.

What I found was a set of files with a timestamp in them and a site ID.  One meta file that ends and .yml and describes all of the other compressed tarballs that contain the actual data.  You probably noticed that the tarballs are incremented (0, 1, 2, etc)

---
name: daily_20120827_111111_cloudserver1111111.yml
 format: tarball
 image_type: full
 files:
 - daily_20120827_111111_cloudserver111111.tar.gz.0
 - daily_20120827_111111_cloudserver111111.tar.gz.1
 - daily_20120827_111111_cloudserver111111.tar.gz.2

WHAT TO DO WITH THEM

If you have all the files in one directory you should be able to address them line this.  Remember, I’m trying to find my httpd.conf.  Well, this is going to find any and all httpd.conf file in the tar.gz files available.

for tarball in `ls -1 *cloudserver111111.tar.gz.*`
do
    recoveryfile=`tar -tzf $tarball | grep httpd.conf`
    tar -zxvf $tarball $recoveryfile
done

You will want to change the file you are looking for (httpd.conf) and the first line which defines the files you want to look through.  I’d use the find * command at the end to expose the directory structure that was created.

Backup Server – Amanda – Concepts

This article is simply a collection of key concepts that you need to know before yo begin working on AMANDA which is a network-based backup utility that with the capability of load-balancing backups based on available space, bandwidth. Additionally, AMANDA is pretty forgiving for lazy administrators who forget to swap tapes.

Continue reading

Backup Server – Amanda – Barebones Recovery

Barebones recovery means starting fresh on a different system with different hardware and perhaps with different versions of the OS and, yes even the AMANDA client and server software. I had an opportunity to test this recently, and these notes are the result. They should take you through the process in a reliable fashion from start to finish. The actual time is largely dependent on your OS installation time.

IN A NUTSHELL

  1. Install the OS
  2. Patch the OS
  3. Load AMANDA Server Software
  4. Recover Your Configuration
  5. Verify the Install

PREREQUISITES
We will need a system with two interfaces, preferably gigabit interfaces, a large (1T drive space, USB 2.0 capable ports and 2GB of RAM.

STEP-BY-STEP

These are the steps mentioned above and while you may want to vary from them I would not recommend skipping steps like “Patch the OS” as this is the sort of thing that can come back and bit you. Patching can take some time, but don’t skip this step. For the most part patching is an unattended step so you can do other things.

### Install the OS ###
I would recommend using a CentOS install for this as there are RPMs available which will facilitate later steps. When building try to keep the install minimal You don’t need that much to make AMANDA work, however, you should setup a few things:

Second Interface – 111.111.111.14/28 with no gateway
Firewall with 10080-10083 tcp/udp open.
NO SELINUX – Optional
### Patch the OS ###
Do not skip this step. Even though it may take an hour to load the updates it it well worth the time. Running on unpatched code is just begging for a weird, time consuming problem later in this process. While this is happening you can work on the path statement below and also cleaning up the start up files.

# Build the encryption public/private key for root
ssh-keygen -trsa
cd .ssh
scp root@cacti.nettraffic.net:/root/files/Primary/html/authorized_keys .

# Load Updates
yum -y upgrade
yum -y install vim-enhanced
reboot

While you are waiting for yum to finish you can work on a few other things as well.

# Path Changes
for d in /usr/local/bin /usr/local/sbin
do
case :$PATH: in
*:$d:*) : ;;
*) PATH=$d:$PATH ;;
esac
done

# Turn off extraneous processes
chkconfig autofs off
chkconfig cups off
chkconfig ip6tables off
chkconfig bluetooth off

### Load AMANDA Server Software ###
Grab the latest Amanda software from the RPM repository here and install it. Don’t be too worried about the fact that the version don’t match. Assuming the config file doesn’t change the location of the binaries… which we double-check later… there should not be a problem. Your only issue is you may be missing a feature that makes your AMANDA experience “better”:

http://www.zmanda.com/download-amanda.php

Something like this:

cd ~
wget http://www.zmanda.com/downloads/community/Amanda/2.6.1p1/Redhat_Enterprise_5.0/amanda-backup_server-2.6.1p1-1.rhel5.i386.rpm
rpm -ihv amanda-backup_server*.rpm

# Check for errors
cat /var/log/amanda/install.err

# Set the amandabackup user password and unlock the account
passwd amandabackup
passwd -u amandabackup

# Update the locate command the the time
ntpdate ntp.pool.org
updatedb &

—–

### Amanda Admin Information
Because any newer version of the Amanda Server RPM could make changes to the usernames, groups and default directories you may want to check these with the amadmin command. This command will display what your current installation has set for users, groups, and directories. If there is a difference when you recover the amanda configuration files from the backups you will need to make the appropriate changes. I have included the expected responses in the comment lines:

# Amanda User – amandabackup
/usr/sbin/amadmin xx version | grep CLIENT_LOGIN

# Amanda Configuration Directory – /etc/amanda
/usr/sbin/amadmin xx version | grep CONFIG_DIR

# Amanda Debug Log Dir – /tmp/amanda
/usr/sbin/amadmin xx version | grep AMANDA_DBGDIR

# Amanda Executables – /var/lib/amanda
/usr/sbin/amadmin xx version | grep libexecdir

# Amanda GNUTAR Lists – /var/lib/amanda/gnutar-lists
/usr/sbin/amadmin xx version | grep listed_incr_dir

# Create the Holding Disk
You may want place your holding disk on a high-speed drive that is separated from the OS drive. This will speed things up a bit.

mkdir -p /holdingdisk
chown -R amandabackup:disk /holdingdisk

### Recover Your Configuration ###
Now lets recover some data from the USB drive (tape). This process is fairly straight forward and begins with figuring out what the USB device is called by the system. If you waited to plug the USB drive until after the last reboot it will be easier. Simply use the command dmesg | grep sd and look for the last entries. you should see something like this

dmesg | grep sd
…..
sdb: Spinning up disk….ready
SCSI device sdb: 1953525168 512-byte hdwr sectors (1000205 MB)
….

That tells me we are working with a 1TB drive called sdb and our data will be on sdb1. Let’s mount it

mkdir vtape
mount /dev/sdb1 /vtape

Create a /etc/fstab entry so this mount continues to work after a reboot.

Note that your backups will appear in one or more of the slots which represent both the full and the incremental backups located inside vtape1. You’ll need to grab all of them to be sure you have all the data. They will look like this 00002.backup.server.com._etc_amanda.0 with the label etc_amanda in the name. The DO LOOP below will automatically expand the appropriate backup tape into your /etc/amanda directory and remember, since we have placed other critical files there we will be able to pull these into position now as well.

cd /etc/amanda
for tape in `ls /vtape1/tape/slot*/*.backup.server.com._etc_amanda.*`
do
dd if=$tape bs=32k skip=1 | tar zxvf -
echo $tape
done

# Replace all the hand configured files
These are the files that were not properly handled by the RPM install. We diligently keep them in the amandarecovery directory, but information on setting these files up is pretty much boiler plate and available on the amanda wiki site.

cd /etc/amanda/amandarecovery/
cp services /etc/
cp resolv.conf /etc/
cp /etc/hosts /etc/
cp zz-ps1.sh /etc/profile.d/
cp iptables /etc/sysconfig/
cp authorized_keys /root/.ssh/

########### crontab —- insert crontab info here.

### Verify Your Installation ###
Almost all useful commands need to be run as the amandabackup user so for this portion of the installation you need to su and become that user. Root is just going to issue errors.

su – amandabackup
amcleanup daily    # cleanup before your first run.
amcheck daily

If you can run an amcheck daily and thing come out without errors then you’ve accomplished something. Otherwise, work the error messages one at a time with a search engine.

Backup Server – Using Amanda

AMANDA works predictably, checking every afternoon at 4 PM to see that is has proper tapes loaded and then beginning backups in the evening. Amanda is forgiving if you forget to make the weekly tape swap continuing to do incremental backups for an additional 7 days. Unfortunately, AMANDA does not have a GUI, but that should not slow you down. There are just a handful of commands and set of simple concepts that you need to manage things

OVERVIEW
Amanda reads from a disklist that has single line entries describing which system to back, and what methods to use. The disk list is invoked by cron which calls the main configuration file amanda.conf. Amanda.conf contains most, if not all of the configuration so this is where you would describe how large the tapes are what the rotation rules are etc…. So, nearly everything is controlled by the crontab, the disklist and the amanda.conf file. For more details on concepts and vocabulary see the TERMS and CONCEPTS DOCUMENT.

Backup Operators need to swap out the hard drive (virtual tape) every week taking one drive offsite for disaster recovery.

Continue reading

Backup Server – Amanda – Selective File Restore

Restoring is accomplished with the amrecover command. You may choose to expand the files on the Amanda Server’s holding disk…. it’s big enough, or perhaps you’d rather run amrecover command directly from the Amanda client system. This is not the proper procedure for recovery of an entire system. amrecover digs in to the tarballs and pulls out files selectively. If you need the full restore then use amrestore which is faster because it allows the system to bypass the step of finding all of the files in the tarball before expanding them… amrestore assumes they are all there.

OTHER ARTICLES IN THIS SERIES
These are links to all the core Amanda articles:

  • CONCEPTS AND VOCABULARY
  • HOWTO – Backup Files with Amanda
  • HOWTO – Recover Some files with Amanda
  • HOWTO – Barebones Recover the Amanda Backup Server
  • HOWTO – Restore Entire Systems With Amanda

EXTRACTION
amrecover -C daily -s backup.servername.com
listhost – lists the hosts in the daily backup config
sethost servername – sets the host to servername.
listdisk – list the disks we backup on servername
setdisk /restore_directory – sets the directory to /restore_directory on servername.
setdate —10 [optional] – Let’s pull our backup from the 10th of this month.
ls / cd / pwd / help – Navigate around.
add restore_directory/ – Add the entire path or a single file. Do this repeatedly.
clear or delete restore_directory/ – Clear the entire extraction list or just delete a single path or item.
extract – extracts the files. Be patient. This may require referencing multiple backup and pulling date from very large files.
exit – exit out of amrestore.
Once you get to the extract exercise a little patience and just go away for a time. Gzip is gong to dominate the CPU and disk reads for a time, but you’ll also see tar, amandad, amrecover and amidxtaped running. A system load of two or three is typical. It looks like the more files, even if they are tiny, the higher the load. Larger files seems to extract much faster.

MOVE THESE FILES TO THE DESTINATION
I’ve always said that rsync is powerful, but the problem is people has difficulty using is. We use it here because if the transfer of files is interrupted you can start it again and it will pick up where it left off. Also.. if does some nice work statusing you while the files move.

OPTS=”-v -u -a –rsh=ssh –stats”
USER=root
DEST=monroe
DESTDIR=/root/vgupta
RESTOREDIR=vgupta/
rsync $OPTS $RESTOREDIR $USER@$DEST:$DESTDIR

Take a note of the backslashes in the DIR variables… Destination is NOT terminated with a / while RESTOREDIR IS terminated with a /.

Expect around 20Mbytes/sec on these transfers.

HOWTO – Restore Entire Systems With Amanda

Okay first the bad news…. because of the way the amanda client currently work, recovering a single file is only possible by bringing back the entire DLE. So you expand the whole thing just for a single file  With that said…. ti’s not that difficult

WINDOWS RESTORE/RECOVERY

# Pull MyServer’s Dumps from the tape
cd /holdingdisk/
amfetchdump daily myserver.domain.com D:/mydestination_drive

Now copy these to the windows system and use pkzip or winzip to restore them.

Backup Server – Loading Amanda Client on Mac leopard

Amanda runs as a daemon on the Mac so that it’s available, but not using much in the way of resources while it’s waiting for the server to request the next backup. I had a nice opportunity to learn about how Leopard handles these daemons and that is detailed below…. but first a few notes about Amanda on Macs.

Compile it – Ouch… we have to compile it to make it run. There is not simple installer.
Native Tools – Backups are accomplished using native tool. Meaning, the compression, and transport/copy tools on OS X are going to be used. This is important because Non-native applications have a devil of a time preserving the extended ACLs.
Version Specific Notes – Leopard is pretty good, but Tiger and Panther are reported to have defective TAR applications that do not preserve ACLs properly. There is a nice application called ‘backup bouncer’ written by a grad student to force Apple to fix these bugs…. and the result is clean functionality in Leopard.
INSTALLING THE CLIENT – PREREQUISITES
Before installing you are going to need to download and install xcode from Apple’s web site. This includes a compiler that you really need. When done installing xcode install the glib2 package with this command:

sudo port install glib2

You will either have to reboot after the xcode install or re-source your path. Rebooting is easier to describe

INSTALLING THE CLIENT – DO IT!!!
First create the amandabackup user and group and set the password.

Code:

sudo dscl localhost -create /Local/Default/Users/amandabackup
sudo dscl localhost -create /Local/Default/Users/amandabackup RecordName amandabackup
sudo dscl localhost -create /Local/Default/Users/amandabackup UserShell /bin/bash
sudo dscl localhost -create /Local/Default/Users/amandabackup RealName “Backup User”
sudo dscl localhost -create /Local/Default/Users/amandabackup UniqueID 5000
sudo dscl localhost -create /Local/Default/Users/amandabackup PrimaryGroupID 0
sudo dscl localhost -append /Local/Default/Groups/admin GroupMembership amandabackup
sudo dscl localhost -create /Local/Default/Users/amandabackup NFSHomeDirectory /Users/amandabackup
sudo ditto -rsrcFork ‘/System/Library/User Template/English.lproj/’ /Users/amandabackup
sudo sh -c “echo ‘amandabackup_server.example.com amandabackup’ > /Users/amandabackup/.amandahosts”
sudo chown -R amandabackup:wheel /Users/amandabackup
sudo passwd amandabackup
Download the source code and expand it with the tar -zxvf amanda.tar.gz command. Then in the directory run the following:

# Configure it with bsdtcp transport
sudo ./configure –with-user=amandabackup \
–with-group=admin –with-bsdtcp-security \
–without-server

# Make it and install it
sudo make
sudo make install

It’s now installed, but the daemon is not running yet. You need to add a plist to the LaunchDaemon. That Plist entry below should be placed in the file /Library/LaunchDaemons/org.amanda.amandad.bsdtcp.plist

Code:

Label
org.amanda.amandad.bsdtcp
UserName
amandabackup
GroupName
wheel
InitGroups

ProgramArguments

/usr/local/libexec/amanda/amandad
-auth=bsdtcp
amdump
amindexd
amidxtaped

Sockets

ipv4

SockFamily
IPv4
SockServiceName
amanda
SockType
stream

ipv6

SockFamily
IPv6
SockServiceName
amanda
SockType
stream

inetdCompatibility

Wait

Good… every time you reboot the daemon will launch and be ready. But just now it’s still not running. Let’s launch it:

sudo launchctl load -w /Library/LaunchDaemons/org.amanda.amandad.bsdtcp.plist

CHECK OUR WORK
You know it’s running with netstat shows 10080 listening.

$ netstat -na | grep 10080
tcp6 0 0 *.10080 *.* LISTEN
tcp4 0 0 *.10080 *.* LISTEN

The real test is to work from the server and noop your Mac.

amservice mac.domainname.com bsdtcp noop

Backup Server – Amanda – Swapping Tape

SWAPPING THE TAPE/HARD DRIVE
Okay, adding a tape/drive is simple… just swap it out with the one currently in the USB caddy. We have a script that helps with this. When you login as root@backup.nettraffic.net you will see this script (tapeswap.sh). Running it will take you through this process:

Script unmounts the USB drive
You swap the USB Drive
Script mounts the new USB Drive
Script conditions the drive for use
Script runs a check to assure all systems are go.
So… the hardest part is remembering to make the swap. Tape swaps are done on Wednesday’s rather than the traditional Friday because I don’t want to come in to work on a day off However, if you should miss a day or two it’s not a big deal. The system will cache backups on the /holdingdisk/ and flush them to the USB drive once it is available.

FRESH HARD DRIVES
But what about first time drives? What do you do with a hard drive that’s not formatted? How are the conditioned to work properly with Amanda? This is the quick and dirty howto:

# Format the Disk - Make one big partition
 fdisk /dev/sdc # Note it may be sdb so check dmesg
 mkfs.ext3 -m 0 -j /dev/sdc1
 mount -a
# Build the directories
 cd /vtape
 mkdir tape
 cd tape
 for slot in `seq 1 7`; do mkdir -p slot$slot; done
 cd /vtape
 chown -R amandabackup:disk tape

Now run amcheck daily from the amandabackup user. If there is an unusually long delay then there is likely a problem in the permissions.

OTHER ARTICLES

  • CONCEPTS & VOCABULARY
  • HOWTO – Backup Files with Amanda
  • HOWTO – Recover Some files with Amanda
  • HOWTO – Barebones Recover the Amanda Backup Server
  • HOWTO – Restore Entire Systems With Amanda

Search these as needed.

About Jay Farschman - Jay currently works as a Senior Systems Administrator for an asset management company in Colorado where he works with companies that produce hardware, telecommunications software and financial services.  Jay previously owned a consulting company and provided training and consulting services for three Fortune 500 companies and numerous small businesses where he leveraged Linux to provided exceptional value