Create and use a local mirrored yum repository for CentOS 6.4 x86_64

A local mirrored yum repository is a copy of the standard yum repositories for CentOS that exists on your local network. It is kept up to date by synchronizing to an external site. If you have more than a few local hosts, using a local mirrored repo can significantly reduce internet traffic and speed up the update process.

This blog describes how to create, sync, test and use a local mirrored repo for CentOS 6.4. You will be shown how to create the repo, keep it in sync using cron, test the implementation and how to update the /etc/yum.repos.d directory on each host. It also provides 4 bash scripts that you can customize for your site to make maintenance easier.

Contents

Introduction

When you type “yum update” on a standard CentOS installation your site is updated from an external site that is randomly chosen from a list of mirrored repositories on the internet. The packages that need to be updated are detected by comparing version information from your site with the version information from the external site and then downloaded through the internet.

For a small number of hosts this works really well because most packages are small and there normally only a few that are updated at a time but once your network grows to more that a small number of hosts, keeping them all in sync can result in a fair amount of internet traffic.

Please note that the term “a small number of hosts” is deliberately vague. Some sites with a slow internet connection would probably see significant performance improvements once they reached 3 or 4 hosts whereas others with a very fast, fat pipe might not see any improvement until they had dozens of hosts. In my case I decided to switch over when I had about 20.

To reduce the internet traffic overhead associated with updating multiple hosts on your network, you can create a local mirrored repository that acts just like an external repository (which is, itself, mirrored). If you have a large site with a fast, fat internet pipe you should consider becoming an external repository to help the community.

This blog describes how to create the local repository, how to keep it synchronized, how to test it and how to update the /etc/yum.repos.d directories on your hosts to make it all work.

Site Configuration Parameters

The first thing that you need to do is determine where you want the local repository to reside. It must be a network resource that is visible to all of the hosts that want to use it. I use an NFS mount. For the purposes of this discussion we will assume that the network path to the mirrored repository is “/shared/repo/“.

Second you need to choose an administrative host that can create/update the repo directories and run the cron job to keep it in sync. For this discussion we will use the host “bigbob“.

Finally you need to determine which repositories you want to mirror. That decision is going to be site dependent but for me I chose the centos_6.4_extras, centos_6.4_os, centos_6.4_updates and elrepo_el6 repositories. The repositories with the “centos_” prefix. The repository with the “elrepo_el6” prefix is from the elrepo repository. It contains lots of useful stuff that is not in the standard release.

To see the repositories that you reference by default, take a look at the “/etc/yum.repos.d/*repo files on your system.

Here is a summary of the site configuration:

Network Path /shared/repo
Admin Host bigbob
Admin Path /shared/repo/bin
CentOS 5.4 Repos extras, os, updates
Other Repos elrepo
Conf Script /shared/repo/bin/repo_conf.sh
Sync Script /shared/repo/bin/repo_sync.sh
Test Script /shared/repo/bin/repo_test.sh
Update Script /shared/repo/bin/repo_update.sh

Configuration and Synchronization Scripts

To configure your site, log into bigbob (your administrative host) and make sure that you have the createrepo package installed. You can do this by running the “yum info createrepo” to see if it is installed. If it is, then you are good to go. If it isn’t then run “yum install -y createrepo“.

After that create the /shared/repo/bin directory so that you have a place to store the scripts and make sure the /shared/repo is available as an NFS mount (or automount). I am not going to describe how to create an NFS mount or automount for hosts on your site. It is a complex topic and there are many great resources available on the web.

repo_conf.sh

Now you need to create the site configuration file. This is a simple bash script that is used by the other scripts to get basic configuration information such as where the repo is located.

repo_sync.sh

Finally you need to create the synchronization script. This script also creates the mirror.

You will need to update the list variable at line 55 to change the external reference to an rsync site near you.

You will also need to update the excludes variable at line 66 to define which packages you wish to exclude. By default all packages are included.

Initialize Your Mirrored Repository

Run the /shared/repo/bin/repo_sync.sh script to create the local mirrored repository. When this script finishes the following top level directories will have been created.

/shared/repo/centos The CentOS 6.4 repositories.
/shared/repo/elrepo The elrepo repositories.
/shared/repo/yum.repos.d The local.repo configuration for each host.

At this point the mirrored repository only exists on disk. It is not used.

Test Your Mirrored Repository

Before putting the mirrored repository into production we need to test it. That is done by running the following test script to create dummy local installation and then running the “yum list” command.

It works by creating /shared/repo/test directory that contains a local yum configuration file and then running the yum list command with that information: “yum -c yum.conf list“.

repo_test.sh

Cron the Synchronization

The repository that you are mirroring will change periodically as bug fixes are released and packages are enhanced. When those changes occur, you need to make sure that they are reflected at your site. You do this by creating a cron job that periodically run the /shared/repo/bin/repo_sync.sh script.

0 22 * * * cd /shared/repo/bin && repo_sync.sh 2>cron.err 1>cron.out

As you can see, I do this once per day but that is probably overkill. You probably only need to do it weekly.

Please note that some sites lock down the package versions that are available to avoid the risk of instability so they would not do this.

Use Your Mirrored Repository

At this point you have a fully functioning mirrored repository but no one is using it. To use it on a host you to do the following:

  1. Log onto the host.
  2. Create a backup of the /etc/yum.repos.d directory.
  3. Remove all of the files in /etc/yum.repos.d directory.
  4. Copy /shared/repo/yum.repos.d/local.repo to /etc/yum.repos.d

repo_update.sh

You can use this script (or a variant) to do this automatically.

This script will automatically make a backup for your and it is re-entrant so it can be run it multiple times without causing problems because it will only update/backup if something has changed.

9 thoughts on “Create and use a local mirrored yum repository for CentOS 6.4 x86_64”

  1. Got to be the best Sys Admin How-To article I’ve ever seen!  Nicely detailed & out of this world scripting.  For a scripting novice, this is an amazing example.

    regarding repo_sync.sh … I changed the external references to these


    list=(
    # OS source & local destination.
    "rsync://mirrors.loosefoot.com/centos/6.4/os/x86_64/"
    "${mirror_repo}/centos/6.4/os/x86_64/"

    # updates source & local destination.
    "rsync://mirrors.loosefoot.com/centos/6.4/updates/x86_64/"
    "${mirror_repo}/centos/6.4/updates/x86_64/"

    # extras source & local destination.
    "rsync://mirrors.loosefoot.com/centos/6.4/extras/x86_64/"
    "${mirror_repo}/centos/6.4/extras/x86_64/"

    # EPEL source & local destination.
    "rsync://mirrors.kernel.org/fedora-epel/6/x86_64/"
    "${mirror_repo}/elrepo/el6/x86_64/"
    )

    I kept everything the same. Ran the script; repos downloaded via rsync; then ran into this:


    # ================================================================
    # createrepo --update /shared/repo/elrepo/el6/x86_64
    # ================================================================
    Could not find valid repo at: /shared/repo/elrepo/el6/x86_64
    Spawning worker 0 with 2997 pkgs
    Spawning worker 1 with 2998 pkgs
    Spawning worker 2 with 2997 pkgs
    Spawning worker 3 with 2998 pkgs

    At this point the script stalls.

    I ^c out, an ran again. repos were up to date, so it got to this point

    createrepo –update /shared/repo/elrepo/el6/x86_64

    … and stalled again.

    Is there a configuration I can clear so I can just run the script again & get past this problem?

    Thanks!

    1. Thank you for your kind comments.

      The problem that you are running into is probably due to the fact that you do not have a shared mount point on your NFS server named “/share“. You can easily verify that by trying “ls /shared/*” or by typing the mount command.

      Your best bet is to find out what your mount points are and then change the mirror_repo variable at line 7 in the repo_conf.sh file to reflect that.

  2. Changed the elrepo reference from rsync://mirrors.thzhost.com/elrepo/elrepo/el6/x86_64/ to rsync://mirrors.neterra.net/elrepo/elrepo/el6/x86_64/ around line 62 of repo_sync.sh because it was generating a protection violation.

  3. Wow! This is comprehensive and nearly bullet proof. I didn’t have to think about this much at all. All I did was edit the location of the repo and the version of CentOS to 6.5. The only oddity that I encountered was that some of the exclusions that I added were ignored. for instance xen4 and fasttrack and SCL” were still downloaded:

    )

    # Packages to exclude.

    excludes=(

    ‘local_centos_6.5_xen4_x86_64’

    ‘local_centos_6.5_SCL_x86_64’

    ‘local_centos_6.5_contrib_x86_64’

    ‘local_centos_6.5_cr_x86_64’

    ‘local_centos_6.5_isos_x86_64’

    ‘local_centos_6.5_fasttrack_x86_64’

    )

    I can’t figure out why, but thank you very much! This saved me a lot of effort.

  4. First of all, this is the best description on this topic I’ve ever seen.

    Second, a typo:

    “Finally you need to do is determine which repositories you want to mirror” ->

    “Finally you need to determine which repositories you want to mirror”

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.