This is Gentoo's testing wiki. It is a non-operational environment and its textual content is outdated.

Please visit our production wiki at https://wiki.gentoo.org

Project:Infrastructure/Mirrors/Wrangling

From Gentoo Wiki (test)
Jump to:navigation Jump to:search

This guide lists the steps needed to properly wrangle new Gentoo mirrors.

Mirror wrangling

Preparing and testing mirrors

1. New mirrors should file a bug under the "Mirrors" product, "New Server" component. If a request comes in via email, ask them to file the request on Bugzilla.

This will help us test out the mirror using our G entoo A utomated R sync D istfiles checker script, or GARD for short.

2. Ask for the following information on the bug, if not provided already:

  • Official mirror name (preferably not your hostname, but the name of your organization)
  • Mirror type: distfiles/releases, gentoo-portage. NOTE: gentoo-portage is always rsync only . Some mirrors may have rsync for distfiles as well. Do not be confused by the difference!
  • URLs: full protocol and path for release/distfile mirrors, hostname or IP for gentoo-portage mirrors
  • Hardware specifications
  • Bandwidth specifications (transfer speed, eg 100Mbit, and any limits you may have)
  • Location
  • Intended concurrent users (for each protocol, mirror type)
  • Admin contact name and email address (This address should be registered in bugzilla so we can CC you to any bug reports)

3. Modify the summary to have the type of mirror and the city as well as country if possible.

4. Build a testing string for GARD. The;: and| are important, make sure they're where they're supposed to be:

CODE String format for GARD
due:YYYY/MM/DD;dist:http://URL%7Cftp://URL%7Crsync://URL (distfiles);portage:FQDN or IP

Here are some string samples:

CODE Server strings
(Everything mirror: distfiles + gentoo-portage)
due:2009/01/01;dist:http://mirror.gentoo.org/gentoo/%7Cftp://mirror.gentoo.org/gentoo/%7Crsync://mirror.gentoo.org/gentoo/;portage:mirror.gentoo.org

(gentoo-portage mirror only)
due:2009/01/01;dist:||;portage:mirror.gentoo.org

(http distfiles mirror only)
due:2009/01/01;dist:http://mirror.gentoo.org/gentoo/%7C%7C;portage:

5. Inform the mirror of testing policies, tell them the due date (when testing will be completed) and give them the following links to make sure their mirror(s) have been setup correctly:

Our policy right now for testing duration is as follows:

  • New distfiles/gentoo-portage mirrors: 2 weeks from date of infra accepting the bug
  • Returning distfiles/gentoo-portage mirrors: Same hardware/IP - 1 week. Different hardware/IP - 2 weeks
  • Problematic mirrors: 1 week to fix the issue, else they're removed from rotation unless we're provided with valid reasons by the admins.
  • Rsync mirrors in any continental rotation on DNS with issues should be removed immediately from those until the issue is resolved. For example, rsync.namerica.gentoo.org , rsync.europe.gentoo.org . You can leave the mirror in its country's rotation.

6. Put the testing string in the Status whiteboard of the bug and mark the bug as IN_PROGRESS. These steps are important for GARD to start checking the mirror.

7. Review the GARD output for issues daily.

8. Report problems back to the mirror ASAP.

9. If there are no problems, try to give a (good) report at the halfway point of the period.

10. Give a final good at the end of the testing period, if everything is good.

Now the mirror is ready to be added officially.

Adding mirrors

For distfiles mirrors

1. Add the mirror to CVS gentoo/xml/htdocs/main/en/mirrors3.xml (check the format of the doc, follow the same).

2. Send the distfiles password email to the new mirror admins, CCmirror-admin@gentoo.org(mirror admins have the password, it is not public).

{{CodeBox|title=Example email|1=Hi, We're glad to have you as our newest source mirror, welcome aboard!

Use this information to sync with our private mirror every 4 hours from Midnight local time. This is for mirror admins only, please do not share this information.

rsync module: masterdistfiles-{eu,us}.gentoo.org::gentoo username: $insert_here password: $insert_here

To make this scriptable, we recommend placing the password in a file and then using the --password-file option with rsync. The file must not be accessible by others (i.e. chmod 640) or rsync will complain. So, as an example, placing the password into '~/distfile_passwd', you would use an rsync command like the following:

rsync -rlptDq --delete -H --password-file=~/distfile_passwd \ $username@masterdistfiles-{eu,us}..gentoo.org::gentoo /your/local/path/goes/here

Please note that we have received several reports of problems when using the -z (compress) option of rsync. Thus, please DO NOT use this option when syncing against this mirror.

Please also make sure you are subscribed to gentoo-mirrors@gentoo.org as we often send out administrative notices and policy changes via this list.

Thanks for helping Gentoo!

Cheers, $name }}

3. Add the mirror/admin-email tothe infra wiki .

4. Put in the following comment and close the bug:

Thanks for helping out Gentoo. I have added your mirror to our list at http://www.gentoo.org/main/en/mirrors2.xml. Please start syncing from masterdistfiles.gentoo.org as the email I have sent to you instructs. We would like you to subscribe to the gentoo-mirrors Mailing List by sending an email to gentoo-mirrors+subscribe@lists.gentoo.org, as we send out notifications of updates, releases, and changes in policy to the mirrors. Please also monitor http://mirrorstats.gentoo.org as this application is used to keep track of the up-to-date nature of our mirrors. Thanks for the hardware and bandwidth!

For gentoo-portage mirrors

1. Add the IP to osprey.gentoo.org , at /home/gmirror/rsync-hosts-allow/hosts.allow.master . Follow the format, re-use existing numbers for a country before adding new ones.

CODE Adding an IP to the ACL
$ ssh osprey.gentoo.org
$ cd /home/gmirror/rsync-hosts-allow
$ ${EDITOR} hosts.allow.master
$ git commit hosts.allow.master

2. To add a new country, just make sure it isn't already there, then create one.

Note
If you've added a new country to this file, chances are countryTLD.gentoo.org doesn't exist, check FIRST and if nothing, create one using the "Add A Domain Page". Email infra@gentoo.org(CC mirror-admin@gentoo.org) to notify them of the new sub-domain.

3. Add the IP into the country (and if needed, continent) DNS rotation(s) in UltraDNS. For example, a new mirror in the US would be rsync(number).us.gentoo.org , so you will add the new IP:

CODE Managing DNS rotation
rsync(number).us.gentoo.org - TTL - 86400

(US/Country rotation)
rsync.us.gentoo.org - TTL - 1800

(Continental rotation)
rsync.namerica.gentoo.org - TTL - 1800
Note
TTLs for rotations are obviously lower, so that we can modify them quickly if needed.

4. Add the mirror to CVS gentoo/xml/htdocs/main/en/mirrors-rsync-data.xml (check the format of the doc, follow the same).

5. Add the mirror/admin-email tothe infra wiki .

6. Put in the following comment (modify as needed) and close the bug:

Thanks for helping out Gentoo. I have added the IP you specified in the bug to our access database, please start syncing to masterportage.gentoo.org per our rsync document at http://www.gentoo.org/doc/en/rsync.xml. We would like you to subscribe to the gentoo-mirrors Mailing List by sending an email to gentoo-mirrors+subscribe@lists.gentoo.org, as we send our notifications of updates, releases, and changes in policy to the mirrors. Once again, thanks for the hardware and bandwidth! Please also monitor http://mirrorstats.gentoo.org/rsync as this application is used to keep track of the up-to-date nature of our mirrors. You are mirror rsync(number).$country.gentoo.org in the rsync.$country.gentoo.org rotation and have been added to the rsync.$continent.gentoo.org rotation as well.

Removing mirrors

For distfiles mirrors

1. Remove the mirror from CVS gentoo/xml/htdocs/main/en/mirrors3.xml (check the format of the doc, follow the same).

2. Remove the mirror/admin-email fromthe infra wiki .

3. Put in the following comment and close the bug: 

Removed $server_name from mirrors listings. Thanks for the hardware and bandwidth donation, your services helped many Gentoo Users. (You have no more obligation to monitor the gentoo-mirrors mailing list.)

For gentoo-portage mirrors

1. Remove the IP from osprey.gentoo.org , at /home/gmirror/rsync-hosts-allow/hosts.allow.master . Leave the existing rsyncX.$CC comment so that it is known that it should be reused.

CODE Removing an IP to the ACL
$ ssh osprey.gentoo.org
$ cd /home/gmirror/rsync-hosts-allow
$ ${EDITOR} hosts.allow.master
$ git commit hosts.allow.master
Note
If you are removing the last mirror in a rsync.$CC rotation, you should talk to infra and make sure there is a solution so users do not try to sync to an dead address.

2. Remove the IP from the country DNS rotation(s) in UltraDNS.

3. Remove the mirror from CVS gentoo/xml/htdocs/main/en/mirrors-rsync-data.xml (check the format of the doc, follow the same).

4. Put in the following comment (modify as needed) and close the bug: 

Removed $server_name from ACLs, removed from DNS. Thanks for the hardware and bandwidth donation, your services helped many Gentoo Users. (You have no more obligation to monitor the gentoo-mirrors mailing list.)

This page is based on a document formerly found on our main website gentoo.org.
The following people contributed to the original document: robbat2, fox2mike, nightmorph
They are listed here because wiki history does not allow for any external attribution. If you edit the wiki article, please do not add yourself here; your contributions are recorded on each article's associated history page.

Retrieved from "https://wikitest.gentoo.org/index.php?title=Project:Infrastructure/Mirrors/Wrangling&oldid=577734"