Documentation of the Apache mirroring system

Mirror Administration
================================

* Receiving Submissions

  Mirrors are submitted to the "Mirrors" component of the
  "Infrastructure" project on the jira issue tracking system according
  to the instructions listed here:
  http://www.apache.org/info/how-to-mirror.html

  Once submitted, they should be checked to make sure they follow our
  guidelines.  Some things to check:
  - Are they properly displaying HEADERs, READMEs and descriptions?
  - Do they have Namewidth=* in their IndexOptions (so full filenames
    can be seen)?
  - Do symlinks work properly?
  - Are they updating regularly (check the DATE file)?
  - Are they using the correct rsync.*.apache.org (Check the SOURCE file)?

* Updating the database

  The mirrors can then be added to the database at
  https://svn.apache.org/repos/asf/infrastructure/site-tools/trunk/mirrors/mirrors.list
  This is a tab-delimitted text file with the following fields:

  type: ftp, http
  country-code: note uk and not gb ; or a 'fake' domain like 'Backup'
  URL: remember the trailing slash
  contact-email 

* Making the database live

  There is now no need to do anything. runmirmon.sh has been updated to
  do an "svn up" on all machines. Committing the mirrors.list file is
  enough.

* Checking for bad mirrors

  Periodically, the list at http://www.apache.org/mirrors/ should be
  checked for mirrors that are innaccessible or are failing to update
  regularly.  There is no need to remove mirrors that go bad for only
  a few days, since the mirror monitoring system will automatically
  prevent them from being used as preferred mirrors.  But once a
  mirror has been gone for a couple weeks, it is unlikely that it is
  coming back.  When removing bad mirrors, make a note in
  https://svn.apache.org/repos/asf/infrastructure/site-tools/trunk/mirrors/STATUS
  and inform the affected mirrors using their contact email address.

Software Used
=============================
1. mirmon
   https://svn.apache.org/repos/asf/infrastructure/site-tools/trunk/mirrors/mirmon/
   http://www.cs.uu.nl/people/henkp/mirmon/

   Checks each mirror in the mirror database for a DATE file
   in their root and keeps a database of how up-to-date is
   each mirror.  Also generates a web page listing mirror
   status that is posted at
   http://www.apache.org/mirrors/

   Requires: perl, wget

2. mirmonmerge.py
   https://svn.apache.org/repos/asf/infrastructure/site-tools/trunk/mirrors/mirmonmerge.py

   Reads the mirror database and writes it out again after merging the
   last update time from the mirmon database and removing the email
   addresses.

   Requires: python

3. runmirmon.sh
   https://svn.apache.org/repos/asf/infrastructure/site-tools/trunk/mirrors/runmirmon.sh

   Run hourly by cron from the apmirror account on the website-hosting
   machine.  This script assumes that
   https://svn.apache.org/repos/asf/infrastructure/site-tools/trunk/mirrors/
   is checked out to /home/apmirror/mirrors/ and that mirmon has been run
   before at least once to prime its database.  It does the following:
   - Updates dist/DATE to the current time.
   - Convert the mirror database to a form mirmon can use
   - Runs mirmon to update its database
   - Runs mirmonmerge.py to update the live database

4. mirrors.cgi
   https://svn.apache.org/repos/asf/infrastructure/site/trunk/docs/dyn/mirrors/

   CGI script that reads the database generated by mirmonmerge.py and
   picks a mirror for downloads based on the country of the client
   and the up-to-dateness of the mirrors.

   Mirrors are eligible to be selected as a preferred mirror if
     - They are HTTP mirrors;
     - They are in the same country as the client; and
     - Their last update time is less than 24 hours

   Clients may also request a specific preferred mirror using the query
   string.  This will be used regardless of status as long as it is one
   of the mirrors in the database.

   Clients may also select a particular update time using the query string.

   If no eligible mirrors are found in the clients country, then the
   clients country will be changed to "US".  If no eligible mirrors
   are found in the US, then a backup mirror is used.

   More details are available at
   http://www.apache.org/dev/mirrors.html

   Requires: python, geoip (C library and python module), ezt python module


Setting Up a New Site
============================

1. Setup an "apmirror" userid and make the home directory can be accessed
at /home/apmirror/

2. sudo -s -u apmirror
   cd /home/apmirror
   svn co https://svn.apache.org/repos/asf/infrastructure/site-tools/trunk/mirrors/

3. Either copy the /home/apmirror/mirrors/mirmon/prod/ directory over from
an existing site, or run mirmon -get "all" to prime the mirmon database.

4. Put the apmirror user in the apbackup group so that it can write to
/www/www.apache.org/dist/DATE (and check the permissions on that file).

5. Download, build, and install the geoip country edition C library and python
binding (to their default locations).  Then copy over /usr/local/etc/GeoIP.conf
from an active site (to get the appropriate key).

6. Setup a cron job to run /home/apmirror/mirrors/runmirmon.sh hourly
from cron, and perhaps /usr/local/bin/geoipupdate monthly.