Applying Yum Package Updates to Multiple Servers Using Fabric

By Ikuya Yamada and Yoshiyasu Takefuji

Introduction

Administrators of multiple servers know that updating software packages can be a tedious task. It requires a log in, followed by the execution of a "yum update" or something similar for each server. Although several tools exist for enabling administrators to update packages automatically, including yum-updatesd and yum-cron, updating the packages without confirmation possibly leads to further issues.

For instance, Yum completely ignores the dependencies of software compiled from the source. If the compiled software has strict version requirements against the update of packages, it may not function properly after the update. Furthermore, even if the package is installed using Source RPM with custom compile options, it is overwritten by a new package and the features specified in the compile options are disabled.

In this article, we introduce a Python-based short script that enables administrators to easily apply command-line package updates to multiple servers. Using this script, the administrator can manually confirm which packages are to be updated. Internally, this script executes a "yum check-update" in sequence in order to verify the presence of package updates and displays a prompt that allows the administrator to select whether the package updates are to be performed.

The proposed script uses a relatively new Python library called Fabric, which allows users to execute commands at multiple remote servers via SSH. Therefore, the script needs to connect to the user's remote servers via SSH.

Although we tested the script on CentOS 5.5 Linux, administrators using other RedHat-based distributions can also use this script.

Example of Error Caused by Yum

In this section, we will provide an actual example for demonstrating the issue that arises in the Yum package update. Postfix, which is a common mail server, implements MySQL support from recent versions. However, the Postfix package provided by the default Yum repository does not include this feature, and hence, many users install Postfix from the Source RPM.

Let us assume that the server has a previous version of Postfix installed from the Source RPM and uses the configuration given in Postfix MySQL Howto .

# sudo rpm -ivh PREVIOUS_POSTFIX_SOURCE_RPM_URL
# cd /usr/src/redhat/
# sudo vim SPEC/postfix.spec
< %define MYSQL 0
> %define MYSQL 1
# rpmbuild -ba SPEC/postfix.spec
# rpm -ivh RPMS/x86_64/postfix-VERSION_NUM.x86_64.rpm

After executing the "yum -y update", Yum detects and installs the newer version of Postfix.

Loaded plugins: downloadonly, fastestmirror
Loading mirror speeds from cached hostfile
(snip)
=======================================================================================================
 Package                   Arch                     Version                     Repository        Size
=======================================================================================================
Updating:
 postfix                   x86_64                   2:2.3.3-2.1.el5_2           base             3.7 M

Transaction Summary
=======================================================================================================
Install       0 Package(s)
Upgrade       1 Package(s)
(snip)
Running Transaction
  Updating       : postfix                                                                         1/2 
warning: /etc/postfix/main.cf created as /etc/postfix/main.cf.rpmnew
  Cleanup        : postfix                                                                         2/2 

Updated:
  postfix.x86_64 2:2.3.3-2.1.el5_2                                                                     

Complete!

Because the installed package does not contain the MySQL support, Postfix starts to output the following errors in /var/log/maillog after it is restarted.

Feb 18 00:31:29 jango postfix/master[29981]: warning: process /usr/libexec/postfix/local pid 6189 exit status 1
Feb 18 00:31:29 jango postfix/master[29981]: warning: /usr/libexec/postfix/local: bad command startup -- throttling
Feb 18 00:32:29 jango postfix/local[6191]: fatal: unsupported dictionary type: mysql

If it is not confirmed whether the updated package works properly, the Yum package update may occasionally result in server failure. Therefore, we recommend the use of our proposed script instead of the automatic package update provided by yum-updatesd and yum-cron .

Installing Fabric

The proposed script requires several software packages. First, the EPEL software repository needs to be installed. Administrators using a Linux distribution based on RedHat Enterprise Linux 5 can install the repository using the following command. Installation instructions can also be found here.

# sudo rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-4.noarch.rpm

Next, several software packages need to be installed.

# sudo yum -y install gcc python26 python26-devel python26-distribute

Now, Fabric can be installed using easy_install , which also installs the required Python libraries using the following command.

# sudo easy_install-2.6 fabric

Script

The proposed script is given below. As Fabric uses fabfile.py as its default file name, the script should be entered into fabfile.py.

# cd INSTALL_DIR
# vim fabfile.py

#!/usr/bin/env python2.6

from fabric.api import env, run, sudo
from fabric.contrib.console import confirm

env.warn_only = True

def update():
    if run("yum check-update").return_code != 0:
        if confirm("execute yum update?", default=False) == True:
            sudo("yum -y update", pty=True)

As mentioned above, the script executes a yum check-update in each remote server and displays a prompt that enables the user to select whether to execute the yum update. Note that env.warn_only is a necessary component; this is because yum returns a non-zero status code if it finds any package updates, which Fabric incorrectly recognizes as an error and responds by shutting the execution down.

Now, the script can be executed. fab is the command that receives the target Python function name as an argument and runs the function. In this case, the function name is update , and thus, the command is as follows.

# fab -H HOST_NAME1,HOST_NAME2 update

After executing the above command, the script checks for the presence of package updates and outputs the following. As can be seen, if package updates exist, the user can choose whether to perform the updates after confirming the packages that are to be updated.

[qui-gon.ikuya.net] Executing task 'update'
[qui-gon.ikuya.net] run: yum check-update
[qui-gon.ikuya.net] out: Loaded plugins: fastestmirror
[qui-gon.ikuya.net] out:
[qui-gon.ikuya.net] out: bind-libs.x86_64                        30:9.3.6-4.P1.el5_5.3            updates
(snip)
[qui-gon.ikuya.net] out: udev.x86_64                             095-14.21.el5_5.1                updates

Warning: run() encountered an error (return code 100) while executing 'yum check-update'

execute yum update? [y/N]

The user name and password can be optionally specified using -u and -p options, respectively.

# fab -u USER_NAME -p PASSWORD -H HOST_NAME1,HOST_NAME2 update

The target remote server addresses can also be specified using the env.hosts variable in fabfile.py.

#!/usr/bin/env python2.6

from fabric.api import env, run, sudo
from fabric.contrib.console import confirm

env.warn_only = True

env.hosts = [
    "HOST_NAME1",
    "HOST_NAME2"
]

def update():
    if run("yum check-update").return_code != 0:
        if confirm("execute yum update?", default=False) == True:
            sudo("yum -y update", pty=True)

Using the above script, the user can run the update without specifying the remote server addresses.

# fab update

Conclusion

This article proposed a short script that enables server administrators to easily apply package updates to remote servers. The script uses a Python library called Fabric , and executes package updates in sequence.

Note that Fabric is a generic Python library and can be used to automate numerous other server administration tasks. If you want to learn more about Fabric, please refer to the official documentation .

Share

Talkback: Discuss this article with The Answer Gang

Ikuya Yamada

---

Ikuya Yamada is an entrepreneur and an experienced software engineer. Currently, he is the founder and the CTO of Studio Ousia Inc., a software R&D company founded in 2007 in Tokyo. He is also a senior visiting researcher at the Keio Research Institute at SFC from 2010. Prior to Studio Ousia, he was the CTO of a listed Japanese software company named Fractalist Inc. and previously the founder and the CEO of a software R&D company called Newrong Inc., which was acquired by Fractalist Inc. in 2005. He obtained his B.S. (2006) and M.S. (2010) from Keio University.

Yoshiyasu Takefuji

---

Yoshiyasu Takefuji was heavily involved in developing a unix based color workstation in 1983 at University of South Florida. Recently he has been monitoring three Linux servers to see the behavior of DOS attacks. He is a chair of SecurityExpo in Japan since 2004 and also a chair of OECD TrustE security product evaluation committee chair in Japan, and advisor of Japan Network Security Association and CMU in Japan.