Project:Portage/Sync

Starting with Portage version 2.2.16, Portage now has a modular sync system. This new system will ease code maintenance as well as make it possible for third party modules to be installed. Many of the users will be learning about the repos.conf style changes that have been in Portage for several years now for the first time. Portage has been migrating away from the PORTDIR and PORTDIR_OVERLAY variable settings for quite some time. Those settings are very limited in their capabilities as compared to the repos.conf style of repository configuration.

The new plug-in sync system is the next step in that migration. Using PORTDIR and PORTDIR_OVERLAY could only list where the repository was location. Users could not specify other important attributes about that repository. The repos.conf style configuration allows for settings to be added on a per sync-type basis. Along with this new expandability and the plug-in sync system, it will be much easier to change to new repository syncing methods.

General help

Note
While Portage can handle repos.conf as either a file or a directory of files. The preferred method is to be used as a directory. Other tools like layman and mirrorselect require and expect it to be a directory. Layman creates and manages its own layman.conf file to register installed overlays with Portage. mirrorselect looks for repos.conf/gentoo.conf file in order to modify the sync-uri parameter for the gentoo repository.

Changes

• /etc/portage/repos.conf/*
  • New auto-sync setting for all repository types (needed):
    auto-sync = yes # default, Possible values are: yes, no, true, false
  • New for git sync-type:
    sync-depth = n where n = {0,1,2,3,...}
    • 0 is equivalent to full git history
    • 1 shallow clone, only current state (default if option is absent)
    • 2, 3, ... history depth to download.
  • New sync-type modules:
    • sync-type = svn
      Sync a subversion repository
    • for <=sys-apps/portage-2.2.18 sync-type = websync
    • for >=sys-apps/portage-2.2.19 sync-type = webrsync
      Perform an emerge-webrsync operation on the repo. By using this sync-type, a user can do a normal emerge --sync or emaint sync -a instead of running the emerge-webrsync command.
    • sync-type = laymansync
      (if installed) runs layman's code to sync the repository (overlay)
  • New native portage postsync hooks
    • /etc/portage/postsync.d/*
      Runs all hook scripts found once, after all repos have been synced. Compatible with all existing postsync.d scripts. Replaces the portage-utils installed post_sync hook script.
    • /etc/portage/repo.postsync.d/*
      Runs each hook script found with three arguments: repo name, sync-uri, location
      Each script is run at the completion of each repository that was synced. Using these arguments, the hook script can decide if it needs to run or just exit. This is Useful for when only a certain repo has been updated and needs to trigger some other process or update system.

Migration

The primary reason for upgrading from make.conf style PORTDIR (deprecated, only there for backward compatibility with old tools, the repos.conf setting is used for its value) and PORTDIR_OVERLAY (still functional, but deprecated) listings and the new repos.conf style definitions is functionality. The repos.conf method is a lot more flexible and powerful for controlling the repositories. The PORTDIR_OVERLAY listings are simple listing which are not capable of listing the options to control the new sync features and abilities, let alone other portage capabilities such as priority.

Portage configuration

If the /etc/portage/repos.conf directory does not exist:

root #mkdir /etc/portage/repos.conf

Note
You only need to do the next step if you wish to change the default settings for the "gentoo" repository

root #cp /usr/share/portage/config/repos.conf /etc/portage/repos.conf/gentoo.conf

Then edit the file for your installation and desired settings, continue as needed with the remaining migration instructions. Edit all repos.conf/*.conf files, add the auto-sync option to each defined repository.

For sync-type, edit it to one of the installed supported types.

Current supported sync types include:

1. rsync
2. git
3. svn
4. webrsync or websync (equivalent to running emerge-webrsync separately).
5. cvs
6. laymansync (if installed by Layman)

FILE /etc/portage/repos.conf/gentoo.conf

[gentoo]
location = /usr/portage
sync-type = rsync
sync-uri = rsync://rsync.gentoo.org/gentoo-portage
auto-sync = yes

Example local overlay sync-able from a git backup:

FILE /etc/portage/repos.conf/my.conf

[The-Pit]
location = /usr/local/portage
sync-type = git
sync-uri = git://me.org/me/The-Pit.git
auto-sync = no

The above overlay can be synced with

user $ emaint sync -r The-Pit

Layman configuration

Manually editing /etc/portage/repos.conf/layman.conf

For an existing repos.conf/layman.conf file:

1) Change/add the sync-type:

FILE /etc/portage/repos.conf/layman.conf

[wtk]
priority = 50
location = /var/lib/layman/wtk
layman-type = git
sync-type = laymansync
sync-uri = git://tremily.us/wtk-overlay.git
auto-sync = yes

2) Ensure Layman 2.3.0 or higher has been installed with the laymansync module.

layman-updater method

Perform the next series of steps when migrating from the old make.conf layman conf_type to repos.conf type.

1) Ensure you have the new plug-in sync Portage version installed:

root #emerge \>=sys-apps/portage-2.2.16

2) Install the new laymansync enabled layman version. To install layman with the portage plugin, you must enable the sync-plugin-portage USE flag:

FILE /etc/portage/package.use

app-portage/layman sync-plugin-portage

root #emerge \>=app-portage/layman-2.3.0

Note
If you have done the repos.conf conversion prior to installing the new plug-in sync Portage and Layman versions (with its laymansync module). Your existing file will be missing the sync-type, sync-uri and auto-sync settings. But don't worry, you only need to skip to step 4 to re-generate it.

Done

Layman uses its own installed database to generate the Portage configuration file.

3) Edit layman.cfg change it to the repos.conf config type:

FILE /etc/layman/layman.cfg

conf_type : repos.conf

Save and exit the file.

4) Generate the new repos.conf/layman.conf file:

root #layman-updater -R

5) Delete layman's make.conf file:

root #rm /var/lib/layman/make.conf

6) Edit /etc/portage/make.conf: remove the source /var/lib/layman/make.conf line.

Done

Operation

Primary control of all sync operations has been moved from emerge to emaint. emerge --sync now runs the emaint sync module with the --auto option. This --auto option performs a sync on only those repositories with the auto-sync setting set to yes' or 'true'. If the auto-sync option is not set to yes or is absent, then emerge --sync may not sync any repositories.

Note
As a result of the default auto-sync = True/Yes setting, commands like eix-sync, esync -l, emerge --sync && layman -S will cause many repositories to be synced multiple times in a row. Please edit your configs or scripts to adjust for the new operation.

Warning
Due to the above default. For any repositories (repos) that you EXPLICITLY do not want to be synced. You MUST set auto-sync = no

The 'emaint sync' module operates similar to layman. It can sync single or multiple repos. It also syncs repos with auto-sync = no set in the respective repos.conf file. This gives users and administrators more flexibility in how and when repositories are synced.

 emaint sync options:
   -a, --auto      Sync auto-sync enabled repos only (includes defaulted repos)
   -A, --allrepos  Sync all repos that have a sync-url defined
   -r REPO, --repo REPO
                   Sync the specified repo
   --sync-submodule {glsa,news,profiles}
                   Restrict sync to the specified
                   submodule(s)  (only rsync supported currently)

Examples

root #emaint sync -a

Equivalent to emerge --sync. Sync all 'auto-sync = true' repos.

Note
Due to the default auto-sync = true setting, this command will sync all repositories that do not have auto-sync = no explicitly set.

root #emaint sync -r foo

Sync the foo repo (ignores auto-sync setting):

root #emaint sync --allrepos

Sync all repositories with a valid sync-type and sync-url defined. (ignores auto-sync setting)

---

Plug-in sync API

Module specifications

Directory structure

/usr/lib/portage/

sync/

modules/

rsync/

__init__.py

rsync.py

git/

__init__.py

git.py

__init__.py

config_checks.py

controller.py

getaddrinfo_validate.py

old_tre_timestamp.py

syncbase.py

Module's __init__.py

Example module __init__.py definition

FILE portage/sync/modules/git/__init__.py

# Copyright 2014 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

"""Git plug-in module for portage.
Performs a git pull on repositories
"""


from portage.sync.config_checks import CheckSyncConfig


module_spec = {
	'name': 'git',
	'description': __doc__,
	'provides':{
		'git-module': {
			'name': "git",
                        'sourcefile': "git",
			'class': "GitSync",
			'description': __doc__,
			'functions': ['sync', 'new', 'exists'],
			'func_desc': {
				'sync': 'Performs a git pull on the repository',
				'new': 'Creates the new repository at the specified location',
				'exists': 'Returns a boolean of whether the specified dir ' +
					'exists and is a valid Git repository',
			},
			'func_parameters': {
				'kwargs': {
					'type': dict,
					'description': 'Standard python **kwargs parameter format' +
						'Please refer to the sync modules specs at ' +
						'"https://wiki.gentoo.org:Project:Portage" for details',
				},
			},
			'validate_config': CheckSyncConfig,
		}
	}
}

Definition: "module_spec"

Field Name	Type	Comments
name	string	The name of the module The name of the file providing the exported classes
description	string	Typically is a pointer to the files docstring
provides : module1 [module2]	dictionary dictionary dictionary	Dictionary of of the different sub modules it provides. Note the field identifiers must be unique, but are not used outside of the plugin system. Unique module identifier Unique module identifier

Definition: module_spec["provides"]["module1"]

Field Name	Type	Comments
name	string	Unique identifier of the exported class.
sourcefile	string	The filename that contains the class without the ".py" extension
class	string	the id/name of the class exported
description	string	Description of the service it performs
functions	list	The list of commands it exports
func_desc	dictionary, its keys are the same as the 'functions' list members above	strings describing each of the exported functions
validate_config	pointer	Assign it the CheckSyncConfig derived class used to validate repos.conf settings for this repo type. For simple modules, the base CheckSyncConfig is suitable. For others please refer to Validating repos.conf variables section of this page

Validating repos.conf variables

Depending on the needs of the sync module, you can assign the "validate_config" variable in the module's spec the portage.sync.config_checks.CheckSyncConfig class or subclass it, adding any checks for additional variables needed to be defined in the repos.conf definition for that sync type.

Example module adding an additional validation test

CODE portage/sync/modules/cvs/__init__.py

# Copyright 2014 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

"""CVS plug-in module for portage.
Performs a cvs up on repositories
"""


from portage.localization import _
from portage.sync.config_checks import CheckSyncConfig
from portage.util import writemsg_level


class CheckCVSConfig(CheckSyncConfig):

	def __init__(self, repo, logger):
		CheckSyncConfig.__init__(self, repo, logger)
		self.checks.append('check_cvs_repo')


	def check_cvs_repo(self):
		if self.repo.sync_cvs_repo is None:
			writemsg_level("!!! %s\n" %
				_("Repository '%s' has sync-type=cvs, but is missing sync-cvs-repo attribute")
				% self.repo.name, level=self.logger.ERROR, noiselevel=-1)


module_spec = {
	'name': 'cvs',
	'description': __doc__,
	'provides':{
		'cvs-module': {
			'name': "cvs",
			'class': "CVSSync",
			'description': __doc__,
			'functions': ['sync', 'new', 'exists'],
			'func_desc': {
				'sync': 'Performs a cvs up on the repository',
				'new': 'Creates the new repository at the specified location',
				'exists': 'Returns a boolean of whether the specified dir ' +
					'exists and is a valid CVS repository',
			},
			'func_parameters': {
				'kwargs': {
					'type': dict,
					'description': 'Standard python **kwargs parameter format' +
						'Please refer to the sync modules specs at ' +
						'"https://wiki.gentoo.org:Project:Portage" for details',
					'required-keys': ['options', 'settings', 'logger', 'repo',
						'xterm_titles', 'spawn_kwargs'],
				},
			},
			'validate_config': CheckCVSConfig,
		}
	}
}

Common classes used by sync modules

• class SyncBase(object):........ Base Sync class meant to be subclassed. It defines some common functions which can be used or overriden as needed by the target module. It provides stubs for all functions needed by the main sync controller.
  • def name():
  • def can_progressbar(self, func):
  • def __init__(self, bin_command, bin_pkg):
  • def _kwargs(self, kwargs):.... Sets internal variables from kwargs
  • def exists(self, **kwargs):.... Tests whether the repo actually exists
  • def sync(self, **kwargs):...... Stub only: Sync the repository
  • def post_sync(self, portdb, location, emerge_config):.... Stub only: repo.sync_type == "Blank" NOTE: Do this after reloading the config, in case it did not exist prior to sync, so that the config and portdb properly account for its existence.

• class NewBase(object):........ Base Sync class which subclasses SyncBase and also meant to be subclassed. It adds a predefined sync(), and stubs for new() and update(). It forms the base for a module which requires separate new and update operations.
  • def sync(self, **kwargs):...... Conditionally syncs the repository using either the new or update functions
  • def new(self, **kwargs):....... Stub only: Do the initial download and install of the repository
  • def update(self):............... Stub only: Update existing repository

• class CheckSyncConfig(object):..... Base repos.conf settings checks class
  • def __init__(self, repo=None, logger=None):.... Class init function
    • self.checks = ['check_uri', 'check_auto_sync']...... List of predefined checks to perform. When subclassing append to it or redfine as needed
  • def repo_checks(self):......................... Perform all checks available
  • def check_uri(self):.............................. Check the sync_uri setting
  • def check_auto_sync(self):................. Check the auto_sync setting