This post is password protected. To view it please enter your password below:

Password:

Share on Facebook

DISK FAULTS
If you have a bad sector or two you can address this with the commands below.  Or, you may need to boot into the live CD and fsck -y

sudo touch /forcefsck
sudo shutdown -r now

SLOW CONTROLLER HARDWARE RESPONSE
If the controller hardware takes too long to respond back with the correct devices the system will advance without properly identifying them.  In Ubuntu you can address this in the /etc/default/grub file by changing the GRUB_CMDLINE_LINUX line as follow:

GRUB_CMDLINE_LINUX=”rootdelay=60″
update-grub
reboot

IMPROPERLY FORMATTED GRUB
Not sure the best way to go about this, but what I read was related to a mismatch in the selected boot partitions in /etc/fstab and those in the grub.  I have definately seen SuperMicro system where the drives flipped about on boot.  This is caused by a RAID configuration that works well with Windows installs but confounds Linux.  It’s controlled at BIOS/CMOS level and can be removed with dmraid.  Yeck!  Drop me a note if you need help with this.  I likely have the recipe for fixing it.

KERNEL PANIC – Missing Init

Init is process id #1, the #1 that all other processes are started from.  The first one run after boot and the one that cleans up when you are shutting down.  This isn’t a kernel panic at all, but a failure on the init process.  FSCK it.

Ubuntu Live CD, opened the terminal and typed:

sudo debugfs -w /dev/sda1
debugfs 1.41.11 (14-Mar-2010)
debugfs:

type in: clri <8>
and after hitting enter,
type this: quit
then reboot. It should force an fsck and come up happy.

Share on Facebook

So, what makes it so difficult? It’s larger than the maximum allowed RPM size of 2GB, but that shouldn’t stop you from breaking it up into two RPMS where the second one calls the installer in silent mode.

I have it working, but I just wondered why the Internet is so silent about this subject.

ANSWER:

A couple of weeks later and it looks like a matter of too much oracle documentation.  Oracle owns the SEO on these terms and penetrating them isn’t going to a simple matter of setting up an teaser article with a few terms filled out.  I’m going to try a few keywords and see how that goes for me.

keywords: rpmbuild

Share on Facebook

• L0 – Intel “Meron” Xeon Core 2
• L1 – Intel “Penryn” Xeon 45nm Core 2
• L2 – Intel “Nehalem” Xeon Core i7
• L3 – Intel “Westmere” Xeon 32 nm Core i7
• L4 – Intel “Sandy Bridge”
• L5 – Intel “Ivy Bridge”

So… just use VMWare’s Compatibility Guide? No, you need to apply some common sense as well.  I just went shopping for an r720 to work with our ‘Sandy-Bridge” hexcore r710′s and found that the compatibility guides doesn’t really cover new systems very well.  Take a look at Wikipedia and look up your processor.  In my case, I have a choice of quite a few Xeon E5-26xx processors, but there is a note that the Xeon E5-2603 and Xeon E5-2609 do not have Hyper-threading capabilities.  Well, my other systems certainly do.  Also, these are quadcore rather than hexcore.  So, problem avoided.  I need to spend a couple hundred more on a proper processor so I don’t have to degrade my EVC Mode.

Reference:

• http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003212
• http://www.vmware.com/resources/compatibility/search.php
• http://en.wikipedia.org/wiki/List_of_Intel_Xeon_microprocessors

About Jay Farschman - Jay currently works as a Senior Systems Administrator for an asset management company in Colorado where he works with companies that produce hardware, telecommunications software and financial services.  Jay previously owned a consulting company and provided training and consulting services for three Fortune 500 companies and numerous small businesses where he leveraged Linux to provided exceptional value.

Share on Facebook

This post is password protected. To view it please enter your password below:

Password:

Share on Facebook

Like a good relational database, PostgreSQL maintains a set of transactional log file known as write-ahead-logs (WAL) in the pg_xlog directory.  These logs are written to for every change in the database files and are used to recovery from a crash condition.  If you crash, replay all the WAL files since the last backup and you will be back in business right at the point of failure.

Well, if you have this capability, what about keeping a warm-standby system and feeding it all the WAL files.  If you teach it how to continuously process the incoming write ahead logs from the live system you will have a system ready to go at a moments notice.  When you read about this setup in other places on line the primary server is known as ‘master’ and the secondary the ‘slave’.

NOTE BENE: Both your primary and your secondary need to be running the same major version of the postgreSQL database.

THE CONTINUOUS FEED

If you’ve read other articles on this blog you see that rsync and ssh tunneling are one of my favorite tools for system administration.  We will be setting up a trust between the live primary server and the warm-backup secondary.  There are other ways to do this, but I’m going to show you the RHEL/CentOS method.

In a nutshell, create an ssh cert on both system and share them with the other system.  So do these command on both system so you have reciprocal trust:

su - postgres
ssh-keygen -trsa
cd ~/.ssh/
cp id_rsa.pub authorized_key

At this point you are authorized to connect as the postgres user to your own machine.  We want to be able to connect the primary and secondary together.  That means having a copy of the id_rsa.pub data in the authorized_key file on the remote system.  So, make that edit and then test the connection

ssh postgres@remotesystem

Almost ready to move on, but in a future step we are going to need the rsync command on both systems.  So, make sure you have it available

yum install rsync

SUDOERS FOR STOP/START/STATUS

Edit your /etc/sudoers file on both systems and make an entry at the bottom so that postgres is allowed to workk with the initialization file on the remote system.

postgres ALL=NOPASSWD: /etc/init.d/postgresql

That works, move on to the fun part.

BUILDING PG_STANDBY

Before we can get this working we will need a special module called pg_standby which does not come standard and probably needs to be compiled.  That should be an obstacle for you.

# If yo don't have the dev tools get them
yum groupinstall "Development Tools"

# Get the source http://www.postgresql.org/ftp/source/
wget http://ftp.postgresql.org/pub/source/v8.4.6/postgresql-8.4.6.tar.gz
tar -zxvf postgresql-8.4.6.tar.gz 
cd postgresql-8.4.6
./configure
gmake all
cd pg_standby
gmake all
cp pg_standby/pg_standby /usr/local/bin/pg_standby
chmod 755 /usr/local/bin/pg_standby

So, we should be good with a new pg_standby.

CONFIGURING POSTGRESQL.CONF ON BOTH SIDES

On the live, primary server edit the archive_ settings so that that are on and able to rsync to the secondary, warm-standby.  Don’t restart or reload the server yet and these setting will not take affect.

archive_mode = on               # allows archiving to be done
archive_command = ‘rsync -a %p postgres@warmstandby:/var/lib/pgsql/walfiles/%f’
archive_timeout = 30           # 30 is a test number.  In service set it to 0

On the secondary, warm-standby create a recovery.conf file in the postgres $HOME directory

su - postgres
mkdir -pv /var/lib/pgsql/walfiles
vi recovery.conf
# this is your line for recovery.conf
restore_command = '/usr/local/bin/pg_standby -l -d -s 2 -t /tmp/pgsql.trigger.5432 /var/lib/pgsql/walfiles %f %p %r 3>>standby.log

Clean up the data directory.  We are very close to replicating with an rsync command:

rm -rf /var/lib/pgsql/data/

Back on the master, let’s put the database in safe-copy mode and make a copy of the live data over to the warm-standby

psql -U postgres -c "SELECT pg_start_backup('dhreplication');"
rsync -azv /var/lib/pgsql/data/* postgres@warmstandby:/var/lib/pgsql/data/
psql -U postgres -c "SELECT pg_stop_backup();"
service postgresql reload

Now, we need to edit the postgresql.conf on the secondary, warm-standby.  This is the file we just copied over in the rsync.

# Turn archive_mode = off
vi /var/lib/pgsql/data/postgresql.conf

Still on the secondary let’s build a symbolic link that places a copy of the recovery.conf file that we created above in the data directory

ln -s /var/lib/pgsql/recovery.conf /var/lib/pgsql/data/recovery.conf

LETS DO IT

Let’s grab three different terminal sessions on the secondary, warm-standby server and do the following on the three terminals:

tail -f /var/lib/pgsql/data/pg_log/postgresql*.log

watch -n1 ls -lah /var/lib/pgsql/walfiles/

/etc/init.d/postgresql start

Your tail command will produce lines line this:

running restore        : OK

removing "/var/lib/pgsql/walfiles/00000001000000030000005B"
LOG:  restored log file "00000001000000030000005E" from archive
Trigger file         : /tmp/pgsql.trigger.5432
Waiting for WAL file    : 00000001000000030000005F
WAL file path        : /var/lib/pgsql/walfiles/00000001000000030000005F
Restoring to        : pg_xlog/RECOVERYXLOG
Sleep interval        : 2 seconds
Max wait interval    : 0 forever
Command for restore    : cp "/var/lib/pgsql/walfiles/00000001000000030000005F" "pg_xlog/RECOVERYXLOG"
Keep archive history    : 00000001000000030000005D and later
WAL file not present yet. Checking for trigger file...
WAL file not present yet. Checking for trigger file...
WAL file not present yet. Checking for trigger file...

The watch command will show you the a new WAL file arriving every 30 seconds.  That may be a bit frequent for you, but for testing it’s nice to not have to wait for the files to move. edit the archive_timeout to change this. The watch command will show something like this

Every 1.0s: ls -lah /var/lib/pgsql/walfiles/                                                                   Mon Sep 10 09:25:34 2012

total 64M
drwxr-xr-x  2 postgres postgres 4.0K Sep 10 09:25 .
drwx------  7 postgres postgres 4.0K Sep 10 07:43 ..
-rw-------  1 postgres postgres  16M Sep 10 09:10 00000001000000030000005B
-rw-------  1 postgres postgres  16M Sep 10 09:15 00000001000000030000005C
-rw-------  1 postgres postgres  16M Sep 10 09:20 00000001000000030000005D
-rw-------  1 postgres postgres  16M Sep 10 09:25 .00000001000000030000005E.7Bu3Vc

TESTING AND CUTOVER

At this point your warm-standby server will stay in sync with the primary, but you won’t be able to use your secondary server.  This is as intended.  Doing a cutover test you want to follow the following steps:

1. Primary – Set the archive_command = /bin/true
2. Primary – service postgresql reload
3. Secondary – touch /tmp/pgsql.trigger.5432
4. Secondary – service postgresql reload

That should do it.

REFERENCE:

http://little418.com/2009/06/setting-up-postgresql-warm-standby-replication-at-joyent.html
http://www.postgresql.org/docs/current/interactive/warm-standby.htm
http://www.postgresql.org/docs/current/interactive/continuous-archiving.html
http://wiki.postgresql.org/wiki/Replication,_Clustering,_and_Connection_Pooling

About Jay Farschman - Jay currently works as a Senior Systems Administrator for an asset management company in Colorado where he works with companies that produce hardware, telecommunications software and financial services.  Jay previously owned a consulting company and provided training and consulting services for three Fortune 500 companies and numerous small businesses where he leveraged Linux to provided exceptional value.

Share on Facebook

That bothered me, but I don’t expect support the guys to be all knowing, even if it is a top-notch organization like Rackspace.  The real answer is yes.  Here is how it’s done.

If you are familiar with the API calls for interacting with RackSpace programmatically, you should probably skip this article, it’s going to be really basic.  If you want to learn these calls, then I found a nice article here that describes pulling and extracting the files for a Windows image and getting a .vhd file

ANATOMY OF A BACKUP

So logging in to the RackSpace Cloud interface and you should see a new(ish) addition to the Hosting Menu.  Choose “Cloud Servers” under the Open Cloud and then you’ll enter a new interface.  Once there click on “Files”  At this point you see your files.  Yes, you can see them in the old interface, but you cannot download them.

What I found was a set of files with a timestamp in them and a site ID.  One meta file that ends and .yml and describes all of the other compressed tarballs that contain the actual data.  You probably noticed that the tarballs are incremented (0, 1, 2, etc)

---
name: daily_20120827_111111_cloudserver1111111.yml

 format: tarball
 image_type: full
 files:
 - daily_20120827_111111_cloudserver111111.tar.gz.0
 - daily_20120827_111111_cloudserver111111.tar.gz.1
 - daily_20120827_111111_cloudserver111111.tar.gz.2

WHAT TO DO WITH THEM

If you have all the files in one directory you should be able to address them line this.  Remember, I’m trying to find my httpd.conf.  Well, this is going to find any and all httpd.conf file in the tar.gz files available.

for tarball in `ls -1 *cloudserver111111.tar.gz.*`
do
    recoveryfile=`tar -tzf $tarball | grep httpd.conf`
    tar -zxvf $tarball $recoveryfile
done

You will want to change the file you are looking for (httpd.conf) and the first line which defines the files you want to look through.  I’d use the find * command at the end to expose the directory structure that was created.

Share on Facebook

TERMINOLOGY:

• Partition - a portion of physical hard disk space. A hard disk may contain one or more partitions. Partitions are defined by BIOS and described by partition tables stored on a harddrive.
• Volume - a logical concept which hides the physical organization of storage space. A compatibility volume directly corresponds to a partition while LVM volume may span more than one partition on one or more physical disks. A volume is seen by users as a single drive letter.
• Physical Volume (PV) Synonym for “hard disk”. A single physical hard drive.
• Volume Group (VG) A set of one or more PVs which form a single storage pool. You can define multiple VGs on each system.
• Logical Volume (LV) A usable unit of disk space within VG. LVs are used analogously to partitions on PCs or slices under Solaris: they usually contain filesystems or paging spaces (“swap”)Unlike physical partition can span multiple physical volumes that constitute VG. .
• Root partition. Physical or logical partition what holds root filesystem and mount points for all other partitions. Can be physical partition or logical volume.

LVM Commands

• pvcreate /dev/hda3 – creates physical volumes
• vgcreate vg01 /dev/hda3 – creates a volume group (in this case, vg01) using the physical volume
• lvcreate -l25000 -nlv01 vg01 – creates a logical volume using the volume group, allocating 25000 blocks
• lvcreate -L4G -nlvroot vgraid1
• vgextend – adds a volume to the volume group (if you add a new disk)
• lvdisplay -v /dev/vg01/lv01
• vgdisplay -v vg01
• lvremove
• vgreduce
• mkfs -t ext3 /dev/vg01/lv02 makes a file system
• mount /dev/vg01/lv02 /home/new mount the file system
• mount -a mounts everything in /etc/fstab
• vgscan –mknodes
• vgchange -a y /dev/vgraid1 (bring /dev/vgraid1 online if it didn’t come up automatically)
• lvscan
• mkfs -t ext4 -t small /dev/vgraid5extra
• umount /home/new (unmount a filesystem)
• pvresize – update size of pv
• lvextend -L+ /dev/vgraid0/lvsharedfiles0 
• e2fsck -f /dev/vgraid1/lvsharedfilestemp (check filesystem)
• resize2fs /dev/vgraid1/lvsharedfilestemp (resize the filesystem) (now works online)

MDADM stuff
RAID
raidtools2
mdadm (probably a better choice)
/etc/raid/raidtab
http://unthought.net/Software-RAID.HOWTO/
http://xtronics.com/reference/SATA-RAID-debian-for-2.6.html
http://juerd.nl/site.plp/debianraid

mdadm -Cv /dev/md0 -l1 -n2 missing /dev/sda1 –auto creates a degraded raid 1 array
mdadm -Cv -c 256 /dev/md20 -l5 -n2 missing /dev/sdf1 array with chunk size

mdadm -A -a /dev/md0 /dev/sda1

mdadm -A -a /dev/md1 /dev/sda2
fixing a degraded array
mdadm –add /dev/md7 /dev/sdd2

mkfs -t ext3 /dev/md20 – make a file system
mkfs -t ext3 -T largefile4 -E stride=16,stripe-width=2 /dev/md20 – reduce inodes, runs faster on big files

largefile4 is defined in /etc/mke2fs.conf, blocksize defaults to 4096, so with chunk size of 256, we need stride of 16

mdadm –detail –scan >> /etc/mdadm/mdadm.conf

Share on Facebook

Something very big will happen in America within the next 180 days. It will be more devastating than the credit crisis of 2008. For most people, it will hit them like a brick wall. It will touch Americans harder and deeper than anything else we've seen since the Great Depression. Michael Lombardi feels so strongly about this, he's decided to present his "Critical Warning Number Six" in a new video. In case you're not familiar with him... Michael Lombardi has been widely recognized as predicting five major economic events over the past 10 years. In 2002, he started advising his readers to buy gold– related investments when gold traded under $300 an ounce. In 2006, he begged his readers to get out of the housing market...before it plunged.

My thoughts, and please share your own on this issue below:

1. Scare Tactic – The voice used for the advertising is a threatening male voice about to tell you something you don’t want to hear.
2. Timing – “Within the next 180-days… “  As of this writing if you poke around on like you can find mention of this ‘prediction’ going back to November of 2011 and possibly further back.
3. Verification of Past Predictions – I tried to use the Internet Archive Wayback Machine to see what this website looked like in the past.  After all, if Mr. Lombardi has posted his past predictions online we should be able to see them.  Unfortunately, the Wayback Machine doesn’t seem to have an old web archives of his content.
4. Who Else Made These Predictions? – I read a book prior to the present economic crisis by Harry S. Dent that seems to say the same thing Michael Lombardi is saying, but the information is verifiable.  A few differences.  The Wayback machine shows Dent ready to publish his book back in July of 2008.   Dent sells a newsletter that essentially draws parallels to the great depression 80-some-years ago.  It’s a good read and I’m a firm believer that history is a fine indicator of the future.  But as the immortal Samuel Clemen once said, “History does not repeat itself, but it does rhyme”
5. Why Share Your Knowledge? – If you lived in the future, let’s say the year 2022 and you traveled back to 2012 what would you do with your knowledge?  How would you leverage this knowledge? One way would be to setup a newsletter and charge $200/year sharing your knowledge with others.  The other would be to take your capital and make smart investments.  Then there is the third way; you could invest with your knowledge of the past and then share the knowledge as a newsletter.  If you invest first and have a large enough following you can make money as your followers invest and drive up prices.

I’ll probably come back to this and add as time allows.

About Jay Farschman - Jay currently works as a Senior Systems Administrator for an asset management company in Colorado where he works with companies that produce hardware, telecommunications software and financial services.  Jay previously owned a consulting company and provided training and consulting services for three Fortune 500 companies and numerous small businesses where he leveraged Linux to provided exceptional value.

Share on Facebook

In its simplest form you will dump all the information to the screen

dmidecode

but that’s a bit much so try running with the -t argument which lets you narrow down the search to the components (bios, system, baseboard, chassis, processor, memory, cache, connector, slot)  So, for instance, if need to learn how much RAM you system can handle:

# dmidecode -t memory
# dmidecode 2.10
SMBIOS 2.7 present.
# SMBIOS implementations newer than version 2.6 are not
# fully supported by this version of dmidecode.

Handle 0x0027, DMI type 16, 23 bytes
Physical Memory Array
    Location: System Board Or Motherboard
    Use: System Memory
    Error Correction Type: Single-bit ECC
    Maximum Capacity: 32 GB
    Error Information Handle: No Error
    Number Of Devices: 4

Enjoy and let me know you you end up using this command.

Share on Facebook