pkgcore Documentation
Release trunk
Brian Harring, Marien Zwart, Tim Harder
October 25, 2014
Contents
1
API Documentation
1.1 Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3
3
2
Man Pages
2.1 Installed Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7
7
3
Developer Notes
3.1 Content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9
9
4
Indices and tables
55
i
ii
pkgcore Documentation, Release trunk
Contents:
Contents
1
pkgcore Documentation, Release trunk
2
Contents
CHAPTER 1
API Documentation
1.1 Modules
pkgcore
pkgcore.binpkg
pkgcore.binpkg.remote
pkgcore.binpkg.repo_ops
pkgcore.binpkg.repository
pkgcore.binpkg.xpak
pkgcore.cache
pkgcore.cache.errors
pkgcore.cache.flat_hash
pkgcore.cache.fs_template
pkgcore.cache.metadata
pkgcore.config
pkgcore.config.basics
pkgcore.config.central
pkgcore.config.cparser
pkgcore.config.dhcpformat
pkgcore.config.domain
pkgcore.config.errors
pkgcore.config.mke2fsformat
pkgcore.const
pkgcore.ebuild
pkgcore.ebuild.atom
pkgcore.ebuild.restricts
pkgcore.ebuild.conditionals
pkgcore.ebuild.const
pkgcore.ebuild.cpv
pkgcore.ebuild.digest
pkgcore.ebuild.domain
pkgcore.ebuild.ebd
pkgcore.ebuild.ebuild_built
pkgcore.ebuild.ebuild_src
pkgcore.ebuild.eclass_cache
pkgcore.ebuild.errors
pkgcore.ebuild.filter_env
pkgcore.ebuild.formatter
Continued on next page
3
pkgcore Documentation, Release trunk
Table 1.1 – continued from previous page
pkgcore.ebuild.misc
pkgcore.ebuild.portage_conf
pkgcore.ebuild.processor
pkgcore.ebuild.profiles
pkgcore.ebuild.repo_objs
pkgcore.ebuild.repository
pkgcore.ebuild.resolver
pkgcore.ebuild.triggers
pkgcore.fetch
pkgcore.fetch.base
pkgcore.fetch.custom
pkgcore.fetch.errors
pkgcore.fs
pkgcore.fs.contents
pkgcore.fs.fs
pkgcore.fs.livefs
pkgcore.fs.ops
pkgcore.fs.tar
pkgcore.gpg
pkgcore.log
pkgcore.merge
pkgcore.merge.const
pkgcore.merge.engine
pkgcore.merge.errors
pkgcore.merge.triggers
pkgcore.operations
pkgcore.operations.domain
pkgcore.operations.format
pkgcore.operations.observer
pkgcore.operations.repo
pkgcore.os_data
pkgcore.package
pkgcore.package.base
pkgcore.package.conditionals
pkgcore.package.errors
pkgcore.package.metadata
pkgcore.package.mutated
pkgcore.package.virtual
pkgcore.pkgsets
pkgcore.pkgsets.filelist
pkgcore.pkgsets.glsa
pkgcore.pkgsets.installed
pkgcore.pkgsets.system
pkgcore.plugin
pkgcore.repository
pkgcore.repository.configured
pkgcore.repository.errors
pkgcore.repository.misc
pkgcore.repository.multiplex
pkgcore.repository.prototype
pkgcore.repository.syncable
pkgcore.repository.util
Continued on next page
4
Chapter 1. API Documentation
pkgcore Documentation, Release trunk
Table 1.1 – continued from previous page
pkgcore.repository.virtual
pkgcore.repository.visibility
pkgcore.repository.wrapper
pkgcore.resolver
pkgcore.resolver.choice_point
pkgcore.resolver.pigeonholes
pkgcore.resolver.plan
pkgcore.resolver.state
pkgcore.resolver.util
pkgcore.restrictions
pkgcore.restrictions.boolean
pkgcore.restrictions.delegated
pkgcore.restrictions.packages
pkgcore.restrictions.restriction
pkgcore.restrictions.util
pkgcore.restrictions.values
pkgcore.scripts
pkgcore.scripts.filter_env
pkgcore.scripts.pclone_cache
pkgcore.scripts.pconfig
pkgcore.scripts.pebuild
pkgcore.scripts.pinspect
pkgcore.scripts.pmaint
pkgcore.scripts.pmerge
pkgcore.scripts.pplugincache
pkgcore.scripts.pquery
pkgcore.spawn
pkgcore.sync
pkgcore.sync.base
pkgcore.sync.bzr
pkgcore.sync.cvs
pkgcore.sync.darcs
pkgcore.sync.git
pkgcore.sync.hg
pkgcore.sync.rsync
pkgcore.sync.svn
pkgcore.system
pkgcore.system.libtool
pkgcore.util
pkgcore.util.commandline
pkgcore.util.file_type
pkgcore.util.packages
pkgcore.util.parserestrict
pkgcore.util.repo_utils
pkgcore.vdb
pkgcore.vdb.contents
pkgcore.vdb.ondisk
pkgcore.vdb.repo_ops
pkgcore.vdb.virtuals
pkgcore.version
1.1. Modules
5
pkgcore Documentation, Release trunk
6
Chapter 1. API Documentation
CHAPTER 2
Man Pages
Pkgcore installs a set of scripts for installing/removing packages, and doing various system maintenance related operations. The man pages for each command follow.
2.1 Installed Commands
7
pkgcore Documentation, Release trunk
8
Chapter 2. Man Pages
CHAPTER 3
Developer Notes
These are the original docs written for pkgcore, detailing some of it’s architecture, intentions, and reasons behind
certain designs.
Currently, the docs aren’t accurate- this will be corrected moving forward.
Right now they’re primarily useful from a background-info standpoint.
3.1 Content
3.1.1 Rough TODO
• rip out use.* code from pkgcore_checks.addons.UseAddon.__init__,
core.ebuild.repository
and generalize it into pkg-
• not hugely important, but... make a cpython version of SlottedDict from pkgcore.util.obj; 3% reduction for full
repo walk, thus not a real huge concern atm.
• userpriv for pebuild misbehaves..
• http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/491285 check into, probably better then my crufty
itersort; need to see how well heapqu’s nlargest pop behaves (looks funky)
• look into converting MULTILIB_STRICT* crap over to a trigger
• install-sources trigger
• recreate verify-rdepends also
• observer objects for reporting back events from merging/unmerging cpython ‘tee’ is needed, contact harring for
details. basic form of it is in now, but need something more powerful for parallelization elog is bound to this
also
• Possibly convert to cpython:
– flat_hash.database._parse_data
– metadata.database._parse_data
– posixpath (os.path)
• get the tree clean of direct /var/db/pkg access
• vdb2 format (ask harring for details).
9
pkgcore Documentation, Release trunk
• pkgcore.fs.ops.merge_contents; doesn’t rewrite the contents set when a file it’s merging is relying on symlinked directories for the full path; eg, /usr/share/X11/xkb/compiled -> /var/blah, it records the former instead
of recording the true absolute path.
• pmerge mods; [ –skip-set SET ] , [ –skip atom ], use similar restriction to –replace to prefer vdb for matching
atoms
• refactor pkgcore.ebuild.cpv.ver_cmp usage to avoid full cpv parsing when _cpv is in use; ‘nuff said, look in
pkgcore.ebuild.cpv.cpy_ver_cmp
• testing of fakeroot integration
it was working back in the ebd branch days; things have changed since then (heavily), enabling/disabling should
work fine, but will need to take a look at the contentset generation to ensure perms/gid leaks through correctly.
• modify repository.prototype.tree.match to take an optional comparison
reasoning being that if we’re just going to do a max, pass in the max so it has the option of doing the initial
sorting without passing through visibility filters (which will trigger metadata lookups)
• ‘app bundles’. Reliant on serious overhauling of deps to do ‘locked deps’, but think of it as rpath based app
stacks, a full apache stack compiled to run from /opt/blah for example.
• pkgcore.ebuild.gpgtree
derivative of pkgcore.ebuild.ebuild_repository, this overloads ebuild_factory and eclass_cache so that gpg
checks are done. This requires some hackery, partially dependent on config.central changes (see above). Need
a way to specify the trust ring to use, ‘severity’ level (different class targets works for me). Anyone who implements this deserves massive cookies.
• pkgcore.ebuild.gpgprofile: Same as above.
• reintroduce locking of certain high level components using read/write; mainly, use it as a way to block sync’ing
a repo that’s being used to build, lock the vdb for updates, etc.
• preserve xattrs when merging files to properly support hardened profiles
• support standard emerge.log output so tools such as qlop work properly
• add FEATURES=parallel-fetch support for downloading distfiles in the background while building pkgs, possibly extend to support parallel downloads
• apply repo masks to related binpkgs (or handle masks somehow)
• remove deprecated PROVIDE and old style virtuals handling
• add argparse support for checking the inputted phase name with pebuild to make sure it exists, currently nonexistent input cause unhandled exceptions
• allow pebuild to be passed ebuild file paths in addition to its current atom handling, this should work similar to
how portage’s ebuild command operates
• support repos.conf (SYNC is now deprecated)
• make profile defaults (LDFLAGS) override global settings from /usr/share/portage/config/make.globals or similar then apply user settings on top, currently LDFLAGS is explicitly set to an empty string in make.globals but
the profile settings aren’t overriding that
• support /etc/portage/mirrors
• support ACCEPT_PROPERTIES and /etc/portage/package.properties
• support ACCEPT_RESTRICT and /etc/portage/package.accept_restrict
• support pmerge –info (emerge –info workalike), requires support for info_vars and info_pkgs files from profiles
10
Chapter 3. Developer Notes
pkgcore Documentation, Release trunk
3.1.2 Changes
(Note that this is not a complete list)
• Proper env saving/reloading. The ebuild is sourced once, and run from the env.
• DISTDIR has indirection now. It points at a directory, ie, symlinks. to the files. The reason for this is to prevent
builds from lying about their sources, leading to less bugs.
• PORTAGE_TMPDIR is no longer in the ebuild env.
• (PORTAGE_|)BUILDDIR is no longer in the ebuild env.
• BUILDPREFIX is no longer in the ebuild env.
• AA is no longer in the ebuild env.
• inherit is an error in phases except for setup, prerm, and postrm. pre/post rm are allowed only in order to deal
with broken envs. Running config with a broken env isn’t allowed, because config won’t work; installing with a
broken env is not allowed because preinst/postinst won’t be executed.
• binpkg building now gets the unmodified contents- thus when merging a binpkg, all files are there unmodified.
3.1.3 Commandline framework
Overview
pkgcore’s own commandline tools and ideally also most external tools use a couple of utilities from pkgcore.util.commandline to enforce a consistent interface and reduce boilerplate. There are also some helpers for writing
tests for scripts using the utilities. Finally, pkgcore’s own scripts are started through a single wrapper (just to reduce
boilerplate).
Writing a script
Whether your script is intended for inclusion with pkgcore itself or not, the first things you should write are a commandline.OptionParser subclass (unless your script takes no commandline arguments) and a main function. The OptionParser is a lightly customized optparse.OptionParser, so the standard optparse documentation applies. Differences
include:
• A couple of standard options and defaults are added. Some of this uses __init__.py, so if you override that
(which you will) remember to call the base class (with any keyword arguments you received).
• The “Values” object used is a subclass, with a “config” property that autoloads the user’s configuration. You
should access this as late as possible for a more responsive ui.
• check_values applies some minor cleanups, see the module for details. Remember to call the base method (you
will usually want to do some things here).
The “main” function takes an optparse “values” object generated by your option parser and two pkgcore.util.formatters.Formatter instances, one for stdout and one for stderr. This one should do the actual work your
script does.
The return value of the main function is your script’s exit status. Returning None is the same thing as returning 0
(success).
If you have used optparse before you might wonder why main only receives an optparse values object, not the remaining arguments. This is handled a bit differently in pkgcore: if you handle arguments you should sanity-check them
in check_values and store them on the values object. check_values should always return an empty tuple as second
3.1. Content
11
pkgcore Documentation, Release trunk
argument, either because no arguments were passed or because they were all accepted by check_values. We believe
this makes more sense, since it stores everything learned from the commandline on a single object.
All output has to go through the formatter. If you use “print” directly the formatter will lose track of where it is in
the line, which will cause weird output if you use the “wrap” option of the formatter. The test helpers also rely on all
output going through the formatters.
To actually run your script you call pkgcore.util.commandline.main (do not confuse this with your own script’s main
function, the two are quite different). The simplest (and most common) call is commandline.main({None:
(yourscript.OptionParser, yourscript.main)}). The weird dict is used for subcommands. The
recommended place to put this call is in a tiny script that just imports your actual script module and calls commandline.main. Making your script an actual module you can import means it can be tested (and it can be useful in
interactive python or for quick hacky scripts).
commandline.main takes care of a couple of things, including setting up a reporter for the standard library’s logging
package and swallowing exceptions from the configuration system. It does not swallow any other exceptions your
script might raise (although this might become an option in the future).
check_values and main: what goes where
The idea (as you can guess from the names) is that check_values makes sure everything passed on the commandline
makes sense, but no more than that.
• The best way to report incorrect commandline parameters is by calling error("error message goes
here") on the option parser. You cannot do this from main, since it has no access to the option parser. Please
do not try to print something similar through the err formatter here, shift the code to check_values.
• check_values does not have access to the out or err formatter. The only way it should “communicate” is through
the error (or possibly exit) methods. If you want to produce different kinds of output, do it in main. (it is possible
the option parser will grow a warning method at some point, if this would be useful let us know (file a trac
ticket).
• Use common sense. If it is part of your script’s main task it should be in main. If it changes the filesystem it
should definitely be in main.
Subcommands
The main function recently gained some support for subcommands (which you probably know from most version
control systems). If you find yourself trying to reimplement this kind of interface with optparse, or one similar to
emerge with a couple of mutually exclusive switches selecting a mode (–depclean, –sync etc.) then you should try
using this subcommand system instead.
To use it, simply define a separate OptionParser and main function for every subcommand and use the subcommand
name as the key in the dict passed to commandline.main. The key None used for “no subcommand” can still be used
too, but this is probably not a good idea.
If there is no parser/main pair with the key None and an unrecognized subcommand is passed (including --help)
an overview of subcommands is printed. This uses the docstring of the __main__ function, so put something useful
there. If there is a None parser you should include the valid subcommands in its --help output, since there is no
way to get at commandline.main’s autogenerated subcommand help if a None parser is present.
pwrapper
Because having a dozen of different scripts each just calling commandline.main would be silly pkgcore’s own scripts
are all symlinks to a single wrapper which imports the right actual script based on the sys.argv[0] it is called with
12
Chapter 3. Developer Notes
pkgcore Documentation, Release trunk
and runs it. The script module needs to define either a commandline_commands dict (for a script with subcommands)
or a class called OptionParser and function called main for this to work.
The script used in the source tree also takes care of inserting the right pkgcore package on sys.path. Installed pkgcore
uses a different wrapper without this magic.
If you write a new script that should go into pkgcore itself, use the wrapper. If you maintain it externally and do not
have a lot of scripts, don’t bother duplicating this wrapper system. Don’t bother duplicating the path manipulation
either: if you put your script in the same directory your actual package or module is in (no separate “bin” directory)
and don’t run it as root no path manipulation is required.
Tests
Because additions to the default options pkgcore uses can make your script unrunnable it is critical to have at least
rudimentary tests that just instantiate your parser. Because optparse defaults to calling sys.exit for a parse failure and the pkgcore version also likes to load the user’s configuration files, writing those tests is slightly tricky.
pkgcore.test.scripts.helpers tries to make it easier. It contains a mangle_parser function that takes an
OptionParser instance and makes it raise exceptions instead of exiting. It also contains a mixin with some extra assert methods that check if your option parser and main function have the desired effect on various arguments and
configurations. See the docstrings for more information.
3.1.4 Config use and implementation notes
Using the manager
Normal use
To get at the user’s configuration:
from pkgcore.config import load_config
config = load_config()
main_repo = config.get_default(’repo’)
spork_repo = config.repo[’spork’]
Usually this is everything you need to know about the manager. Some things to be aware of:
• Some of the managed sources of configuration data may be slow, so accessing a source is delayed for as long as
possible. Some things require accessing all sources though and should therefore be avoided. The easiest one to
trigger is config.repo.keys() or the equivalent list(config.sections(‘repo’)). This has to get the “class” setting for
every available config section, which might be slow.
• For the same reason the manager does not know what type names exist (there is no hardcoded list of them, so
the only way to get that information would be loading all config sections). This is why you can get this:
>>> load_config().section(’repo’) # typo, should be "sections"
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: ’_ConfigMapping’ object is not callable
This constructed a dictlike object for accessing all config sections of the type “section”, then tried to call it.
Testcase use
For testing of high-level scripts it can be useful to construct a config manager containing hardcoded values:
3.1. Content
13
pkgcore Documentation, Release trunk
from pkgcore.config import basics, central
config = central.ConfigManager([{
’repo’ = basics.HardCodedConfigSection({’class’: my_repo,
’data’: [’1’, ’2’]}),
’cont’ = basics.ConfigSectionFromStringDict({’class’: ’pkgcore.my.cont’,
’ref’: ’repo’}),
}])
What this does should be fairly obvious. Be careful you do not use the same ConfigSection object in more than one
place: caching will not behave the way you want. See Adding a config source for details.
Adding a configurable
You often do not really have to do anything to make something a valid “class” value, but it is clearer and it is necessary
in certain cases.
Adding a class
To make a class available, do this:
from pkgcore.config import ConfigHint, errors
class MyRepo(object):
pkgcore_config_type = ConfigHint({’cache’: ’section_ref’},
typename=’repo’)
def __init__(self, repo):
try:
self.initialize(repo)
except SomeRandomException:
raise errors.InstantiationError(’eep!’)
The first ConfigHint arg tells the config system what kind of arguments you take. Without it it assumes arguments with
no default are strings and guesses for the other args based on the type of the default value. So if you have no default
values or they are just None you should tell the system about your args.
The second one tells it you fulfill the repo “protocol”, meaning your instances will show up in load_config().repo.
ConfigHint takes some more arguments, see the api docs for details.
Adding a callable
To make a callable available you can do this:
from pkgcore.config import configurable, errors
@configurable({’cache’: ’section_ref’}, typename=repo)
def my_repo(repo):
# do stuff
configurable is just a convenience function that applies a ConfigHint.
14
Chapter 3. Developer Notes
pkgcore Documentation, Release trunk
Exception handling
If you raise an exception when the config system calls you it will catch the exception and wrap it in an InstantiationError. This is good for calling code since catching and printing those provides the user with a readable description of
what happened. It is less good for developers since the raising of a new exception kills the traceback printed in debug
mode. You will have a traceback that “ends” in the config code handling instantiation.
You can improve this by raising an InstantiationError yourself. If you do this the config system will be able to add the
extra information needed for a user-friendly error message to it without raising a new exception, meaning debug mode
will give a traceback leading right back to your code raising the InstantiationError.
Adding a config source
Config sources are pretty straightforward: they are mappings from a section name to a ConfigSection subclass. The
only tricky thing is the combination of section references and caching. The general rule is “do not expose the same
ConfigSection in more than one way”. If you do it will be collapsed and instantiated once for every way it is exposed,
which is usually not what you want. An example:
from pkgcore.config import basics, configurable
def example():
return object()
@configurable({’ref’: ’section_ref’})
def nested(ref):
return ref
multi = basics.HardCodedConfigSection({’class’: example})
myconf = {
’multi’: multi,
’bad’: basics.HardCodedConfigSection({’class’: nested, ’ref’: multi})
’good’: basics.ConfigSectionFromStringDict({’class’: ’nested’,
’ref’: ’multi’})
If you feed this to the ConfigManager and instantiate everything “multi” and “good” will be identical but “bad” will
be a different object. For an explanation of why this happens see the implementation notes in the next section.
You trigger a similar problem if you create a custom ConfigSection subclass that bypasses central’s collapse_named_section for named section refs. If you somehow get at the referenced ConfigSection and hand it to
collapse_section you will most likely circumvent caching. Only use collapse_section for unnamed sections.
ConfigManager tries not to extract more things from this mapping than it has to. Specifically, it will not call
__getitem__ before it needs to instantiate the section or needs to know its type. However it will iterate over the
keys (section names) immediately to find autoloads. If this is a problem (getting those names is slow) then make sure
the manager knows your config is “remote”.
Implementation notes
This code has evolved quite a bit over time. The current code/design tries among other things to:
• Allow sections to contain both named and nameless/inline references to other sections.
• Allow serialization of the loaded config.
• Not do unnecessary work (if possibly not recollapse configs, definitely not trigger unnecessary imports, access
configs unnecessarily, reinstantiate configs)
3.1. Content
15
pkgcore Documentation, Release trunk
• Provide both end-user error messages that are complete enough to track down a problem in a complex nested
config and tracebacks that reach back to actual buggy code for developers.
Overview from load_config() to instantiated repo
When you call load_config() it looks up what config files are available (/etc/pkgcore.conf, ~/.pkgcore.conf,
/etc/make.conf) and loads them. This produces a dict mapping section names to ConfigSection instances. For
the ini-format pkgcore.conf files this is straightforward, for make.conf this is a lot of work done in pkgcore.config.portage_conf. I’m not going to describe that module here, read the source for details.
The ConfigSections have a pretty straightforward api: they work like dicts but get passed a string describing what
“type” the value should be and a central.ConfigManager instance for reasons described later. Passing in this “type”
string when getting the value is necessary because the way things like lists of strings are stored depends on the format
of the configuration file but the parser does not have enough information to know it should parse as a list instead of a
string. For example, an ini-format pkgcore.conf could contain:
[my-overlay-cache]
class=pkgcore.cache.flat_hash.database
auxdbkeys=DEPEND RDEPEND
We want to turn that auxdbkeys value into a list of strings in the ini file parser code instead of in the central.ConfigManager or even higher up because more exotic config sections may want to store this in a different way
(perhaps as a comma-separated list, or even as “<el>DEPEND</el><el>RDEPEND</el>”. But there is obviously not
enough information in the ini file for the parser to know this is meant as a list instead of a string with a space in it.
central.ConfigManager gets instantiated with one or more of those dicts mapping section names to ConfigSections.
They’re split up into normal and “remote” configs which I’ll describe later, let’s assume they’re all “remote” for now.
In that case no work is done when the ConfigManager is instantiated.
Getting an actual configured object out of the ConfigManager is split in two phases. First the involved config sections
are “collapsed”: inherits are processed, values are converted to the right type, presence of required arguments is
checked, etc. Everything up to actually instantiating the target class and actually instantiating any section references
it needs. The result of this work is bundled in a CollapsedConfig instance. Actual instantiation is handled by the
CollapsedConfig instance.
The ConfigManager manages CollapsedConfig instances. It creates new ones if required and makes sure that if a
cached instance is available it is used.
For the remainder of the example let’s assume our config looks like this:
[spork]
inherit=cache
auxdbkeys=DEPEND RDEPEND
[cache]
class=pkgcore.cache.flat_hash.database
Running config.repo[’spork’] runs config.collapse_named_section(‘spork’). This first checks if this section was already collapsed and returns the CollapsedConfig if it is available. If it is not in the cache it looks up the ConfigSection
with that name in the dicts handed to the ConfigManager on instantiation and calls collapse_section on it.
collapse_section first recursively finds any inherited sections (just the “cache” section in this case). It then grabs
the ‘class’ setting (which is always of type ‘callable’). In this case that’s “pkgcore.cache.flat_hash.database”,
which the ConfigSection imports and returns. This is then wrapped in a config.basics.ConfigType. A ConfigType contains the information necessary to validate arguments passed to the callable. It uses the magic
pkgcore_config_type attribute if the callable has it and introspection for everything else. In this case pkgcore.cache.flat_hash.database.pkgcore_config_type is a ConfigHint stating the “auxdbkeys” argument is of type “list”.
16
Chapter 3. Developer Notes
pkgcore Documentation, Release trunk
Now that collapse_section has a ConfigType it uses it to retrieve the arguments from the ConfigSections and passes
the ConfigType and arguments to CollapsedConfig’s __init__. Then it returns the CollapsedConfig instance to collapse_named_section. collapse_named_section caches it and returns it.
Now we’re back in the __getattr__ triggered by config.repo[’spork’]. This checks if the ConfigType on the CollapsedConfig is actually ‘repo’, and returns collapsedConfig.instantiate() if this matches.
Lazy section references
The main reason the above is so complicated is to support various kinds of references to other sections. Example:
[spork]
class=pkgcore.Spork
ref=foon
[foon]
class=pkgcore.Foon
Let’s say pkgcore.Spork has a ConfigHint stating the type of the “ref” argument is “lazy_ref:foon” (lazy reference
to a foon) and its typename is “repo”, and pkgcore.Foon has a ConfigHint stating its typename is “foon”. a “lazy
reference” is an instance of basics.LazySectionRef, which is an object containing just enough information to produce
a CollapsedConfig instance. This is not the most common kind of reference, but it is simpler from the config point of
view so I’m describing this one first.
When collapse_section runs on the “spork” section it calls section.get_value(self, ‘ref:repo’, ‘section_ref’). “lazy_ref”
in the type hint is converted to just “ref” here because the ConfigSections do not have to distinguish between lazy
and “normal” references. Because this particular ConfigSection only supports named references it returns a LazyNamedSectionRef(central, ‘ref:repo’, ‘foon’). This just gets handed to Spork’s __init__. If the Spork decides to call
instantiate() on the LazyNamedSectionRef it calls central.collapse_named_section(‘foon’), checks if the result is of
type foon, instantiates it and returns it.
The same thing using a dhcp-style config:
spork {
class pkgcore.Spork;
ref {
class pkgcore.Foon;
};
}
In this format the reference is an inline unnamed section. When get_value(central, ‘ref:repo’, ‘foon’) is called it
returns a LazyUnnamedSectionRef(central, ‘ref:repo’, section) where section is a ConfigSection instance for the nested
section (knowing just that “class” is “pkgcore.Foon” in this case). This is handed to Spork.__init__. If Spork calls
instantiate() on it it calls central.collapse_section(self.section) and does the same type checking and instantiating
LazyNamedSectionRef did.
Notice neither Spork nor ConfigManager care if the reference is inline or named. get_value just has to return a
LazySectionRef instance (LazyUnnamedSectionRef and LazyNamedSectionRef are subclasses of this). How this
actually gets a referenced config section is up to the ConfigSection whose get_value gets called.
Normal section references
If Spork’s ConfigHint defines the type of its “ref” argument as “ref:foon” instead of “lazy_ref:foon” it gets handed
an actual Foon instance instead of a LazySectionRef to one. This is built on top of the lazy reference code. For
the ConfigSections nothing changes (the same get_value call is made). But the ConfigManager now immediately
3.1. Content
17
pkgcore Documentation, Release trunk
calls collapse() on the LazySectionRef, retrieving a CollapsedConfig instance (for the “foon”). This is handed to the
CollapsedConfig for “spork”, and when this one is instantiated the referenced CollapsedConfig is also instantiated.
Miscellaneous details
The support for nameless sections means neither ConfigSection nor CollapsedConfig have a name attribute. This
makes the error handling code a bit tricky as it has to tag in the name at various points, but this works better than
enforcing names where it does not make sense (means lots of unnecessary duplication of names when dealing with
dicts of HardCoded/StringBasedConfigSections).
The suppport for serialization of the loaded config means section_refs cannot be instantiated straight away. The object
used for serialization is the CollapsedConfig which gives you both the actual values and the type they have. If the
CollapsedConfig contained arbitrary instantiated objects serializing them would be impossible. So it contains nested
CollapsedConfigs instead.
Not doing unnecessary work is done by caching in two places. The simple one is CollapsedConfig caching its instantiated value. This is pretty straightforward. The more subtle one is ConfigManager caching CollapsedConfigs
by name. It is obviously a good idea to cache these (if we didn’t we would have to cache the instantiated value in
the ConfigManager). An alternative would be caching them by their ConfigSection. This has the minor disadvantage
of keeping the ConfigSection in memory, and the larger one that it may break caching for weird config sources that
generate ConfigSections on demand. The downside of caching by name is we have to make sure nothing generates
a CollapsedConfig for a named section in a way other than collapse_named_section (handing the ConfigSection to
collapse_section bypasses caching).
This means a ConfigSection cannot return a raw ConfigSection from a section_ref get_value call. If it was a ConfigSection that central then collapsed and the reference was actually to a named section caching is bypassed.
The need for a section name starting with “autoload” is also there to avoid unnecessary work. Without this we would
have to figure out the typename of every section. While we can do that without entirely collapsing the config we
cannot avoid importing the “class”, which means load_config() would import most of pkgcore. That should definitely
be avoided.
3.1.5 Checking the source out
If you’re just installing pkgcore from a released tarball, skip this section.
To get the current (development) code with history, install git_ (emerge git on gentoo) and run:
git clone git://pkgcore.org/pkgcore
3.1.6 Installing pkgcore
Set PYTHONPATH
If you only want to run scripts from pkgcore itself (the ones in its “bin” directory) you do not have to do anything
with PYTHONPATH. If you want to use pkgcore from an interactive python interpreter session you do not have to do
anything if you start the interpreter from the “root” of the pkgcore source tree. For other uses you probably want to set
PYTHONPATH to include your pkgcore directory, so that python can find the pkgcore code. For example:
$ export PYTHONPATH="${PYTHONPATH}:/home/user/pkgcore/"
Now test to see if it works:
18
Chapter 3. Developer Notes
pkgcore Documentation, Release trunk
$ python -c ’import pkgcore’
Python will scan pkgcore, see the pkgcore directory in it (and that it has __init__.py), and use that.
Registering plugins
Pkgcore uses plugins for some basic functionality. You do not really have to do anything to get this working, but things
are a bit faster if the plugin cache is up to date. This happens automatically if the cache is stale and the user running
pkgcore may write there, but if pkgcore is installed somewhere system-wide and you only run it as user you can force
a regeneration with:
# pplugincache
If you want to update plugin caches for something other than pkgcore’s core plugin registry, pass the package name as
an argument.
Test pkgcore
Drop back to normal user, and try:
$ python
>>> import pkgcore.config
>>> from pkgcore.ebuild.atom import atom
>>> conf=pkgcore.config.load_config()
>>> tree=conf.get_default(’domain’).repos[1]
>>> pkg=max(tree.itermatch(atom("dev-util/diffball")))
>>> print pkg
>>> print pkg.depends
>=dev-libs/openssl-0.9.6j >=sys-libs/zlib-1.1.4 >=app-arch/bzip2-1.0.2
At the time of writing the domain interface is in flux, so this example might fail for you. If it doesn’t work ask for
assistance in #pkgcore on freenode, or email ferringb (at) gmail.com’ with the traceback.
Build extensions
If you want to run pkgcore from its source directory but also want the extra speed from the compiled extension
modules, compile them in place:
$ python setup.py build_ext -i
3.1.7 Ebuild EAPI
This should hold the proposed (with a chance of making it in), accepted, and implemented changes for ebuild format
version 1. A version 0 doc would also be a good idea ( no one has volunteered thus far ).
Version 0 (or undefined eapi, <=portage-2.0.52*)]
Version 1
This should be fairly easy stuff to implement for the package manager, so this can actually happen in a fairly short
timeframe.
• EAPI = 1 required
3.1. Content
19
pkgcore Documentation, Release trunk
• src_configure phase is run before src_compile. If the ebuild or eclass does not override there is a default that does
nothing. Things like econf should be run in this phase, allowing rerunning the build phase without rerunning
configure during development.
• Make the default implementation of phases/functions available under a second name (possibly using EXPORT_FUNCTIONS) so you can call base_src_compile from your src_compile.
• default src_install. Exactly what goes in needs to be figured out, see bug 33544.
• RDEPEND=”${RDEPEND-${DEPEND}}” is no longer set by portage, same for eclass.
• (proposed) BDEPEND metadata addition, maybe. These are the dependencies that are run on the build system
(toolchain, autotools etc). Useful for ROOT != “/”. Probably hard to get right for ebuild devs who always have
ROOT=”/”.
• default IUSE support, IUSE=”+gcj” == USE=”gcj” unless the user disables it.
• GLEP 37 (“Virtuals Deprecation”), maybe. The glep is “deferred”. How much of this actually needs to be done?
package.preferred?
• test depend, test src_uri (or represent test in the use namespace somehow).
TEST_{SRC_URI,{B,R,}DEPEND}, test “USE” flag getting set by FEATURES=test.
Possibilities:
• drop AA (unused).
• represent in metadata if the pkg needs pkg_preinst to have access to ${D} or not. If this is not required a binpkg
can be unpacked straight to root after pkg_preinst. If pkg_preinst needs access to ${D} the binpkg is unpacked
there as usual.
• use groups in some form (kill use_expand off).
• ebuilds can no longer use PORTDIR and ECLASSDIR(s); they break any potential remote, and are dodgey as
all hell for multiple repos combined together.
• disallow direct access to /var/db/pkg
• deprecate ebuild access/awareness of PORTAGE_* vars; perl ebuilds security fix for PORTAGE_TMPDIR
(rpath stripping in a way) might make this harder.
• use/slot deps, optionally repository deps.
• hard one to slide in, but change versioning rules; no longer allow 1.006, require it to be 1.6
• pkg_setup must be sandboxable.
• allowed USE conditional configurations; new metadata key, extend depset syntax to include xor, represent
allowed configurations.
• true incremental stacking support for metadata keys between eclasses/ebuilds; RESTRICT=-strip for example
in the ebuild.
• drop -* from keywords; it’s package.masking, use that instead (-arch is acceptable although daft)
• blockers aren’t allowed in PDEPEND (the result of that is serious insanity for resolving)
Version 1+
Not sure about these. Maybe some can go into version 1, maybe they will happen later.
• Elibs
• some way to ‘bind’ a rdep/pdep so that it’s explicit “I’m locked against the version I was compiled against”
• some form of optional metadata specifying that a binpkg works on multiple arches, iow it doesn’t rely on
compiled components.
20
Chapter 3. Developer Notes
pkgcore Documentation, Release trunk
• A way to move svn/cvs/etc source fetching over to the package manager. The current way of doing this through
an eclass is a bit ugly since it requires write access to the distdir. Moving it to the package manager fixes that
and allows integrating it with things like parallel fetch. This needs to be fleshed out.
3.1.8 Feature (FEATURES) categories
relevant list of features
• autoaddcvs
• buildpkg
• ccache
• collision-protect
• confcache
• cvs
• digest
• distcc
• distlocks
• fixpackages
• getbinpkg
• gpg
• keeptemp
• keepwork
• mirror
• noclean (keeptemp, keepwork)
• nodoc
• noinfo
• noman
• nostrip
• notitles
• sandbox
• severe
• severer (dumb spanky)
• sfperms
• sign
• strict
• suidctl
• test
• userpriv
3.1. Content
21
pkgcore Documentation, Release trunk
• userpriv_fakeroot
• usersandbox
Undefined
fixpackages
Dead
• usersandbox
• noclean
• getbinpkg (it’s a repo type, not a global feature)
• buildpkg (again, repo thing. moreso ui/buildplan execution)
Build
• keeptemp, keepwork, noclean, ccache, distcc
• sandbox, userpriv, fakeroot
• userpriv_fakeroot becomes fakeroot
• confcache
• noauto (fun one)
• test
repos or wrappers
Mutables
• autoaddcvs
• cvs
• digest
• gpg
• no{doc,info,man,strip}
• sign
• sfperms
• collision-protect (vdb only)
Immutables
• strict
• severe ; these two are repository opts on gpg repo class
22
Chapter 3. Developer Notes
pkgcore Documentation, Release trunk
Fetchers
• distlocks, sort of.
3.1.9 Filesystem Operations
Here we define types of operations that pkgcore will support, as well as the stages where these operations occur.
- File Deletion ( Removal )
• prerm
• unmerge files
• postrm
- File Addition ( Installation )
• preinst
• merge files
• postinst
- File Replacement ( Overwriting )
• preinst
• merge
• postinst
• prerm
• unmerge
• postrm
3.1.10 Python Code Guidelines
Note that not all of the existing code follows this style guide. This doesn’t mean existing code is correct.
Stats are all from a sempron 1.6Ghz with python 2.4.2.
Finally, code _should_ be documented, following epydoc/epytext guidelines
Follow pep8, with following exemptions
• <80 char limit is only applicable where it doesn’t make the logic ugly. This is not an excuse to have a 200 char
if statement (fix your logic). Use common sense.
• Combining imports is ok.
• Use absolute imports
• _Simple_ try/except combined lines are acceptable, but not forced (this is your call). example:
3.1. Content
23
pkgcore Documentation, Release trunk
try: l.remove(blah)
except IndexError: pass
• For comments, 2 spaces trailing is pointless- not needed.
• Classes should be named SomeClass, functions/methods should be named some_func.
• Exceptions are classes. Don’t raise strings.
• Avoid __var ‘private’ attributes unless you absolutely have a reason to hide it, and the class won’t be inherited
(or that attribute must _not_ be accessed)
• Using string module functions when you could use a string method is evil. Don’t do it.
• Use isinstance(str_instance, basestring) unless you _really_ need to know if it’s utf8/ascii
Throw self with a NotImplementedError
The reason for this is simple: if you just throw a NotImplementedError, you can’t tell how the path was hit if derivative
classes are involved; thus throw NotImplementedError(self, string_name_of_attr)
This gives far better tracebacks.
Be aware of what the interpreter is actually doing
Don’t use len(list_instance) when you just want to know if it’s nonempty/empty:
l=[1]
if l: blah
# instead of
if len(l): blah
python looks for __nonzero__, then __len__. It’s far faster than if you try to be explicit there:
python -m timeit -s ’l=[]’ ’if len(l) > 0: pass’
1000000 loops, best of 3: 0.705 usec per loop
python -m timeit -s ’l=[]’ ’if len(l): pass’
1000000 loops, best of 3: 0.689 usec per loop
python -m timeit -s ’l=[]’ ’if l: pass’
1000000 loops, best of 3: 0.302 usec per loop
Don’t explicitly use has_key. Rely on the ‘in’ operator
python -m ’timeit’ -s ’d=dict(zip(range(1000), range(1000)))’ ’d.has_key(1999999)’
1000000 loops, best of 3: 0.512 usec per loop
python -m ’timeit’ -s ’d=dict(zip(range(1000), range(1000)))’ ’1999999 in d’
1000000 loops, best of 3: 0.279 usec per loop
Python interprets the ‘in’ command by using __contains__ on the instance. The interpreter is faster at doing getattr’s
than actual python code is: for example, the code above uses d.__contains__ , if you do d.has_key or d.__contains__,
it’s the same speed. Using ‘in’ is faster because it has the interpreter do the lookup.
So be aware of how the interpreter will execute that code. Python code specified attribute access is slower then the
interpreter doing it on its own.
24
Chapter 3. Developer Notes
pkgcore Documentation, Release trunk
If you’re in doubt, python -m timeit is your friend. ;-)
Do not use [] or {} as default args in function/method definitions
>>> def f(default=[]):
>>>
default.append(1)
>>>
return default
>>> print f()
[1]
>>> print f()
[1,1]
When the function/class/method is defined, the default args are instantiated _then_, not per call. The end result of this
is that if it’s a mutable default arg, you should use None and test for it being None; this is exempted if you _know_ the
code doesn’t mangle the default.
Visible curried functions should have documentation
When using the currying methods (pkgcore.util.currying) for function mangling, preserve the documentation via
pretty_docs.
If this is exempted, pydoc output for objects isn’t incredibly useful.
Unit testing
All code _should_ have test case functionality. We use twisted.trial - you should be running >=2.2 (<2.2 results in
false positives in the spawn tests). Regressions should be test cased, exempting idiot mistakes (e.g, typos).
We are more than willing to look at code that lacks tests, but actually merging the code to integration requires that it
has tests.
One area that is (at the moment) exempted from this is the ebuild interaction; testing that interface is extremely hard,
although it _does_ need to be implemented.
If tests are missing from code (due to tests not being written initially), new tests are always desired.
If it’s FS related code, it’s _usually_ cheaper to try then to ask then try
...but you should verify it ;)
existing file (but empty to avoid reading overhead):
echo > dar
python -m ’timeit’ -s ’import os’ ’os.path.exists("dar") and open("dar").read()’
10000 loops, best of 3: 36.4 usec per loop
python -m ’timeit’ -s ’import os’ $’try:open("dar").read()\nexcept IOError: pass’
10000 loops, best of 3: 22 usec per loop
nonexistant file:
rm foo
python -m ’timeit’ -s ’import os’ ’os.path.exists("foo") and open("foo").read()’
10000 loops, best of 3: 29.8 usec per loop
3.1. Content
25
pkgcore Documentation, Release trunk
python -m ’timeit’ -s ’import os’ $’try:open("foo").read()\nexcept IOError: pass’
10000 loops, best of 3: 27.7 usec per loop
As you can see, there is a bit of a difference. :)
Note that this was qualified with “If it’s FS related code”; syscalls are not cheap- if it’s not triggering syscalls, the next
section is relevant.
Catching Exceptions in python code (rather then cpython) isn’t cheap
stats from python-2.4.2
When an exception is caught:
python -m ’timeit’ -s ’d=dict(zip(range(1000), range(1000)))’ $’try: d[1999]\nexcept KeyError: pass’
100000 loops, best of 3: 8.7 usec per loop
python -m ’timeit’ -s ’d=dict(zip(range(1000), range(1000)))’ $’1999 in d and d[1999]’
1000000 loops, best of 3: 0.492 usec per loop
When no exception is caught, overhead of try/except setup:
python -m ’timeit’ -s ’d=dict(zip(range(1000), range(1000)))’ $’try: d[0]\nexcept KeyError: pass’
1000000 loops, best of 3: 0.532 usec per loop
python -m ’timeit’ -s ’d=dict(zip(range(1000), range(1000)))’ $’d[0]’
1000000 loops, best of 3: 0.407 usec per loop
This doesn’t advocate writing code that doesn’t protect itself- just be aware of what the code is actually doing, and be
aware that exceptions in python code are costly due to the machinery involved.
Another example is when to use or not to use dict’s setdefault or get methods:
key exists:
# Through exception handling
python -m timeit -s ’d=dict.fromkeys(range(100))’ ’try: x=d[1]’ ’except KeyError: x=42’
1000000 loops, best of 3: 0.548 usec per loop
# d.get
python -m timeit -s ’d=dict.fromkeys(range(100))’ ’x=d.get(1, 42)’
1000000 loops, best of 3: 1.01 usec per loop
key doesn’t exist:
# Through exception handling
python -m timeit -s ’d=dict.fromkeys(range(100))’ ’try: x=d[101]’ ’except KeyError: x=42’
100000 loops, best of 3: 8.8 usec per loop
# d.get
python -m timeit -s ’d=dict.fromkeys(range(100))’ ’x=d.get(101, 42)’
1000000 loops, best of 3: 1.05 usec per loop
The short version of this is: if you know the key is there, dict.get() is slower. If you don’t, get is your friend. In other
words, use it instead of doing a containment test and then accessing the key.
Of course this only considers the case where the default value is simple. If it’s something more costly “except” will
do relatively better since it’s not constructing the default value if it’s not needed. So if in doubt and in a performancecritical piece of code: benchmark parts of it with timeit instead of assuming “exceptions are slow” or “[] is fast”.
26
Chapter 3. Developer Notes
pkgcore Documentation, Release trunk
cpython ‘leaks’ vars into local namespace for certain constructs
def f(s):
while True:
try:
some_func_that_throws_exception()
except Exception, e:
# e exists in this namespace now.
pass
# some other code here...
From the code above, e bled into the f namespace- that’s referenced memory that isn’t used, and will linger until the
while loop exits.
Python _does_ bleed variables into the local namespace- be aware of this, and explicitly delete references you don’t
need when dealing in large objs, especially dealing with exceptions:
class c:
d = {}
for x in range(1000):
d[x] = x
While the class above is contrived, the thing to note is that c.x is now valid- the x from the for loop bleeds into the
class namespace and stays put.
Don’t leave uneeded vars lingering in class namespace.
Variables that leak from for loops _normally_ aren’t an issue, just be aware it does occur- especially if the var is
referencing a large object (thus keeping it in memory).
So... for loops leak, list comps leak, dependent on your except clause they can also leak.
Do not go overboard with this though. If your function will exit soon do not bother cleaning up variables by hand. If
the “leaking” things are small do not bother either.
The current code deletes exception instances explicitly much more often than it should since this was believed to clean
up the traceback object. This does not work: the only thing “del e” frees up is the exception instance and the arguments
passed to its constructor. “del e” also takes a small amount of time to run (clearing up all locals when the function
exits is faster).
Unless you need to generate (and save) a range result, use xrange
:: python -m timeit ‘for x in range(10000): pass’ 100 loops, best of 3: 2.01 msec per loop
$ python -m timeit ‘for x in xrange(10000): pass’ 1000 loops, best of 3: 1.69 msec per loop
Removals from a list aren’t cheap, especially left most
If you _do_ need to do left most removals, the deque module is your friend.
Rightmost removals aren’t too cheap either, depending on what idiocy people come up with to try and ‘help’ the
interpreter:
python -m timeit $’l=range(1000);i=0;\nwhile i < len(l):\n\tif l[i]!="asdf":del l[i]\n\telse:i+=1’
100 loops, best of 3: 4.12 msec per loop
python -m timeit $’l=range(1000);\nfor i in xrange(len(l)-1,-1,-1):\n\tif l[i]!="asdf":del l[i]’
100 loops, best of 3: 3 msec per loop
3.1. Content
27
pkgcore Documentation, Release trunk
python -m timeit ’l=range(1000);l=[x for x in l if x == "asdf"]’
1000 loops, best of 3: 1 msec per loop
Granted, that’s worst case, but the worst case is usually where people get bitten (note the best case still is faster for list
comprehension).
On a related note, don’t pop() unless you have a reason to.
If you’re testing for None specifically, be aware of the ‘is’ operator
Is avoids the equality protocol, and does a straight ptr comparison:
python -m timeit ’10000000 != None’
1000000 loops, best of 3: 0.721 usec per loop
$ python -m timeit ’10000000 is not None’
1000000 loops, best of 3: 0.343 usec per loop
Note that we’re specificially forcing a large int; using 1 under 2.5 is the same runtime, the reason for this is that it
defaults to an identity check, then a comparison; for small ints, python uses singletons, thus identity kicks in.
Deprecated/crappy modules
• Don’t use types module. Use isinstance (this isn’t a speed reason, types sucks).
• Don’t use strings module. There are exceptions, but use string methods when available.
• Don’t use stat module just to get a stat attribute- e.g.,:: import stats l=os.stat(“asdf”)[stat.ST_MODE]
# can be done as (and a bit cleaner) l=os.stat(“asdf”).st_mode
Know the exceptions that are thrown, and catch just those you’re interested in
try:
blah
except Exception:
blah2
There is a major issue here. It catches SystemExit exceptions (triggered by keyboard interupts); meaning this code,
which was just bad exception handling now swallows Ctrl+c (meaning it now screws with UI code).
Catch what you’re interested in only.
tuples versus lists.
The former is immutable, while the latter is mutable.
Lists over-allocate (a cpython thing), meaning it takes up more memory then is used (this is actually a good thing
usually).
If you’re generating/storing a lot of sequences that shouldn’t be modified, use tuples. They’re cheaper in memory, and
people can reference the tuple directly without being concerned about it being mutated elsewhere.
However, using lists there would require each consumer to copy the list to protect themselves from mutation. So...
over-allocation + allocating a new list for each consumer.
Bad, mm’kay.
28
Chapter 3. Developer Notes
pkgcore Documentation, Release trunk
Don’t try to copy immutable instances (e.g. tuples/strings)
Example: copy.copy((1,2,3)) is dumb; nobody makes a mistake that obvious, but in larger code people do (people even
try using [:] to copy a string; it returns the same string since it’s immutable).
You can’t modify them, therefore there is no point in trying to make copies of them.
__del__ methods mess with garbage collection
__del__ methods have the annoying side affect of blocking garbage collection when that instance is involved in a
cycle- basically, the interpreter doesn’t know what __del__ is going to reference, so it’s unknowable (general case)
how to break the cycle.
So... if you’re using __del__ methods, make sure the instance doesn’t wind up in a cycle (whether careful data structs,
or weakref usage).
A general point: python isn’t slow, your algorithm is
l = []
for x in data_generator():
if x not in l:
l.append(x)
That code is _best_ case O(1) (e.g., yielding all 0’s). The worst case is O(N^2).
l=set()
for x in data_generator():
if x not in l:
l.add(x)
Best/Worst are now constant (this isn’t strictly true due to the potential expansion of the set internally, but that’s
ignorable in this case).
Furthermore, the first loop actually invokes the __eq__ protocol for x for each element, which can potentially be quite
slow if dealing in complex objs.
The second loop invokes __hash__ once on x instead (technically the set implementation may invoke __eq__ if a
collision occurs, but that’s an implementation detail).
Technically, the second loop still is a bit innefficient:
l=set(data_generator())
is simpler and faster.
An example data for people who don’t see how _bad_ this can get:
python -m timeit $’l=[]\nfor x in xrange(1000):\n\tif x not in l:l.append(x)’
10 loops, best of 3: 74.4 msec per loop
python -m timeit $’l=set()\nfor x in xrange(1000):\n\tif x not in l:l.add(x)’
1000 loops, best of 3: 1.24 msec per loop
python -m timeit ’l=set(xrange(1000))’
1000 loops, best of 3: 278 usec per loop
The difference here is obvious.
3.1. Content
29
pkgcore Documentation, Release trunk
This does _not_ mean that sets are automatically better everywhere, just be aware of what you’re doing- for a single
search of a range, the setup overhead is far slower then a linear search. Nature of sets, while the implementation may
be able to guess the proper list size, it still has to add each item in; if it cannot guess the size (ie, no size hint, generator,
iterator, etc), it has to just keep adding items in, expanding the set as needed (which requires linear walks for each
expansion). While this may seem obvious, people sometimes do effectively the following:
python -m timeit -s ’l=range(50)’ $’if 1001 in set(l): pass’
100000 loops, best of 3: 12.2 usec per loop
python -m timeit -s ’l=range(50)’ $’if 1001 in l: pass’
10000 loops, best of 3: 7.68 usec per loop
What’s up with __hash__ and dicts
A bunch of things (too many things most likely) in the codebase define __hash__. The rule for __hash__ is (quoted
from http://docs.python.org/ref/customization.html):
Should return a 32-bit integer usable as a hash value for dictionary operations. The only required property
is that objects which compare equal have the same hash value.
Here’s a quick rough explanation for people who do not know how a “dict” works internally:
• Things added to it are dumped in a “bucket” depending on their hash value.
• To check if something is in the dict it first determines the bucket to check (based on hash value), then does
equality checks (__cmp__ or __eq__ if there is one, otherwise object identity comparison) for everything in the
bucket (if there is anything).
So what does this mean?
• There’s no reason at all to define your own __hash__ unless you also define __eq__ or __cmp__. The behaviour
of your object in dicts/sets will not change, it will just be slower (since your own __hash__ is almost certainly
slower than the default one).
• If you define __eq__ or __cmp__ and want your object to be usable in a dict you have to define __hash__. If
you don’t the default __hash__ is used which means your objects act in dicts like only object identity matters
until you hit a hash collision and your own __eq__ or __cmp__ kicks in.
• If you do define your own __hash__ it has to produce the same value for objects that compare equal, or you get
really weird behaviour in dicts/sets (“thing in dict” returning False because the hash values differ while “thing
in dict.keys()” returns True because that does not use the hash value, only equality checks).
• If the hash value changes after the object was put in a dict you get weird behaviour too (“s=set([thing]);
thing.change_hash();thing in s” is False, but “thing in list(s)” is True). So if your objects are mutable they
can usually provide __eq__/__cmp__ but not __hash__.
• Not having many hash “collisions” (same hash value for objects that compare nonequal) is good, but collisions
are not illegal. Too many of them just slow down dict/set operations (in a worst case scenario of the same hash
for every object dict/set operations become linear searches through the single hash bucket everything ends up
in).
• If you use the hash value directly keep in mind that collisions are legal. Do not use comparisons of hash values
as a substitute for comparing objects (implementing __eq__ / __cmp__). Probably the only legitimate use of
hash() is to determine an object’s hash value based on things used for comparison.
__eq__ and __ne__
From http://docs.python.org/ref/customization.html:
30
Chapter 3. Developer Notes
pkgcore Documentation, Release trunk
There are no implied relationships among the comparison operators. The truth of x==y does not imply that
x!=y is false. Accordingly, when defining __eq__(), one should also define __ne__() so that the operators
will behave as expected.
They really mean that. If you define __eq__ but not __ne__ doing ”!=” on instances compares them by identity. This
is surprisingly easy to miss, especially since the natural way to write unit tests for classes with custom comparisons
goes like this:
self.assertEqual(YourClass(1), YourClass(1))
# Repeat for more possible values. Uses == and therefore __eq__,
# behaves as expected.
self.assertNotEqual(YourClass(1), YourClass(2))
# Repeat for more possible values. Uses != and therefore object
# identity, so they all pass (all different instances)!
So you end up only testing __eq__ on equal values (it can say “identical” for different values without you noticing).
Adding a __ne__ that just does “return not self == other” fixes this.
__eq__/__hash__ and subclassing
If your class has a custom __eq__ and it might be subclassed you have to be very careful about how you “compare” to
instances of a subclass. Usually you will want to be “different” from those unconditionally:
def __eq__(self, other):
if self.__class is not YourClass or other.__class__ is not YourClass:
return False
# Your actual code goes here
This might seem like overkill, but it is necessary to avoid problems if you are subclassed and the subclass does not
have a new __eq__. If you just do an “isinstance(other, self.__class__)” check you will compare equal to instances of a
subclass, which is usually not what you want. If you just check for “self.__class__ is other.__class__” then subclasses
that add a new attribute without overriding __eq__ will compare equal when they should not (because the new attribute
differs).
If you subclass something that has an __eq__ you should most likely override it (you might get away with not doing
so if the class does not do the type check demonstrated above). If you add a new attribute don’t forget to override
__hash__ too (that is not critical, but you will have unnecessary hash collisions if you forget it).
This is especially important for pkgcore because of pkgcore.util.caching. If an instance of a class with a broken __eq__
is used as argument for the __init__ of a class that uses caching.WeakInstMeta it will cause a cached instance to be
used when it should not. Notice the class with the broken __eq__ does not have to be cached itself to trigger this!
Getting this wrong can cause fun behaviour like atoms showing up in the list of fetchables because the restrictions
they’re in compare equal independent of their “payload”.
Exception subclassing
It is pretty common for an Exception subclass to want to customize the return value of str() on an instance. The easiest
way to do that is:
class MyException(Exception):
"""Describe when it is raised here."""
def __init__(self, stuff):
Exception.__init__(self, ’MyException because of %s’ % (stuff,))
3.1. Content
31
pkgcore Documentation, Release trunk
This is usually easier than defining a custom __str__ (since you do not have to store the value of “stuff” as an attribute)
and you should be calling the base class __init__ anyway.
(This does not mean you should never store things like “stuff” as attrs: it can be very useful for code catching the
exception to have access to it. Use common sense.)
Memory debugging
Either heappy, or dowser are the two currently recommended tools.
To use dowser, insert the following into the code wherever you’d like to check the heap- this is blocking also:
import cherrpy
import dowser
cherrypy.config.update({’engine.autoreload_on’: False})
try:
cherrypy.quickstart(dowser.Root())
except AttributeError:
cherrypy.root = dowser.Root()
cherrypy.server.start()
For using heappy, see the heappy documentation in pkgcore/dev-notes.
3.1.11 resolver
Current design doesn’t coalesce- expects that each atom as it’s passed in specifies the dbs, which is how it does it’s
update/empty-tree trickery.
This isn’t optimal. Need to flag specific atoms/matches as “upgrade if possible” or “empty tree if possible”, etc;
via this, we get coalescing behaviour. Specifically, if the targets are git[subversion] and subversion, we want both
upgraded. So when resolving git[subversion] and encountering dev-util/subversion, we should aim for upgrading it
per the commandline request.
Additional question- should we apply this coalescing awareness to intermediate atoms along the way resolution wise?
specifically, the cnf/dnf solutions, grabbing those and stating “yeah, collapse to these if possible since they’re likely
required” ?
3.1.12 resolver redesign
Hate to say it, but should go back to a specific ‘resolve’ method w/ the resolver plan object holding targets- reason
being, we may have to backtrack the whole way.
3.1.13 config/use issues
need to find a way to clone a stack, getting a standalone config stack if possible for the resolver- specifically so it
can do resets as needed, track what is involved (use dep forcing) w/out influencing preexisting access to that tree, nor
being affected by said usage.
3.1.14 hardlink merge
no comments, just need to get around to it.
32
Chapter 3. Developer Notes
pkgcore Documentation, Release trunk
3.1.15 How to use guppy/heapy for tracking down memory usage
This is a work in progress. It will grow a bit and it may not be entirely accurate everywhere.
Tutorial of sorts
All this was done on a checkout of [email protected], you should be able to
check that out and follow along using something like:
bzr revert -rrevid:[email protected]
in a pkgcore branch.
Heapy is powerful but has a learning curve.
Problems are the documentation (http://guppype.sourceforge.net/heapy_Use.html among others) is a bit unusual and there are various dynamic importing
and other tricks in use that mean things like dir() are less helpful than they are on more “normal” python objects. This
document’s main purpose is to show you how to ask heapy various kinds of questions. It may or may not show a few
cases where pkgcore uses more memory than it should too.
First, get an x86. Heapy currently does not like 64 bit archs much.
Emerge it:
emerge guppy
Fire up an interactive python prompt, set stuff up:
>>>
>>>
>>>
>>>
from guppy import hpy
from pkgcore.config import load_config
c = load_config()
hp = hpy()
Just to show how annoying heapy’s internal tricks are:
>>> dir(hp)
[’__doc__’, ’__getattr__’, ’__init__’, ’__module__’, ’__setattr__’, ’_hiding_tag_’, ’_import’, ’_name
>>> help(hp)
Help on class _GLUECLAMP_ in module guppy.etc.Glue:
_GLUECLAMP_ = <guppy.heapy.Use interface at 0x-484b8554>
This object is your “starting point”, but as you can see the underlying machinery is not giving away any useful usage
instructions.
Do everything that allocates some memory but is not the problem you are tracking down now. Then do:
>>> hp.setrelheap()
Everything allocated before this call will not be in the data sets you get later.
Now do your memory-intensive thing:
>>> l = list(x for x in c.repo["portdir"] if x.data)
Keep an eye on system memory consumption. You want to use up a lot but not all of your system ram for nicer
statistics. The python process was eating about 109M res in top when the above stuff finished, which is pretty good
(for my 512mb ram box).
>>> h = hp.heap()
3.1. Content
33
pkgcore Documentation, Release trunk
The fun one. This object is basically a snapshot of what’s reachable in ram (minus the stuff excluded through setrelheap
earlier) which you can do various fun tricks with. Its str() is a summary:
>>> h
Partition of a
Index Count
0 985931
1 24681
2 49391
3 115974
4 152181
5 36009
6 11328
7 24702
8 11424
9 24681
<54 more rows.
set of 1449133 objects. Total
%
Size
% Cumulative %
68 46300932 45 46300932 45
2 22311624 22 68612556 67
3 21311864 21 89924420 88
8 3776948
4 93701368 91
11 3043616
3 96744984 94
2 1584396
2 98329380 96
1 1540608
1 99869988 97
2
889272
1 100759260 98
1
851840
1 101611100 99
2
691068
1 102302168 100
Type e.g. ’_.more’ to view.>
size = 102766644 bytes.
Kind (class / dict of class)
str
dict of pkgcore.ebuild.ebuild_src.package
dict (no owner)
tuple
long
weakref.KeyedRef
dict of pkgcore.ebuild.ebuild_src.ThrowAwayNameSpace
types.MethodType
list
pkgcore.ebuild.ebuild_src.package
(You might want to keep an eye on ram usage: heapy made the process grow another dozen mb here. It gets painfully
slow if it starts swapping, so if that happens reduce your data set).
Notice the “Total size” in the top right: about 100M. That’s what we need to compare later numbers with.
So here we can see that (surprise!) we have a ton of strings in memory. We also have various kinds of dicts. Dicts are
treated a bit specially: the “dict of pkgcore.ebuild.ebuild_src.package” simply means “all the dicts that are __dict__
attributes of instances of that class”. “dict (no owner)” are all the dicts that are not used as __dict__ attribute.
You probably guessed what you can use “index” for:
>>> h[0]
Partition of a set of 985931 objects. Total size = 46300932 bytes.
Index Count
%
Size
% Cumulative % Kind (class / dict of class)
0 985931 100 46300932 100 46300932 100 str
Ok, that looks pretty useless, but it really is not. The “sets” heapy gives you (like “h” and “h[0]”) are a bunch of
objects, grouped together by an “equivalence relation”. The default one (with the crazy name “Clodo” for “Class or
dict owner”) groups together all objects of the same class and dicts with the same owner. We can also partition the sets
by a different equivalence relation. Let’s do a silly example first:
>>> h.bytype
Partition of a
Index Count
0 985931
1 85556
2 115974
3 152181
4 36009
5 24702
6 11424
7 24681
8 11328
9
408
<32 more rows.
set of 1449133 objects. Total
%
Size
% Cumulative %
68 46300932 45 46300932 45
6 45226592 44 91527524 89
8 3776948
4 95304472 93
11 3043616
3 98348088 96
2 1584396
2 99932484 97
2
889272
1 100821756 98
1
851840
1 101673596 99
2
691068
1 102364664 100
1
317184
0 102681848 100
0
26112
0 102707960 100
Type e.g. ’_.more’ to view.>
size = 102766644 bytes.
Type
str
dict
tuple
long
weakref.KeyedRef
types.MethodType
list
pkgcore.ebuild.ebuild_src.package
pkgcore.ebuild.ebuild_src.ThrowAwayNameSpace
types.CodeType
As you can see this is the same thing as the default view, but with all the dicts lumped together. A more useful one is:
>>> h.byrcs
Partition of a set of 1449133 objects. Total size = 102766644 bytes.
Index Count
%
Size
% Cumulative % Referrers by Kind (class / dict of class)
0 870779 60 43608088 42 43608088 42 dict (no owner)
34
Chapter 3. Developer Notes
pkgcore Documentation, Release trunk
1 24681
2 221936
3 242236
4
6
5 36009
2 22311624
15 20575932
17 8588560
0 1966736
2 1773024
6 11328
7 26483
8 11328
9
3
<132 more rows.
22
20
8
2
2
65919712
86495644
95084204
97050940
98823964
64
84
93
94
96
1 1540608
1 100364572 98
2
800432
1 101165004 98
1
724992
1 101889996 99
0
393444
0 102283440 100
Type e.g. ’_.more’ to view.>
pkgcore.ebuild.ebuild_src.package
dict of pkgcore.ebuild.ebuild_src.package
tuple
dict of weakref.WeakValueDictionary
dict (no owner), dict of
pkgcore.ebuild.ebuild_src.package, weakref.KeyedRef
pkgcore.ebuild.ebuild_src.ThrowAwayNameSpace
list
dict of pkgcore.ebuild.ebuild_src.ThrowAwayNameSpace
dict of pkgcore.repository.prototype.IterValLazyDict
What this does is:
• for every object, find all its referrers
• Classify those referrers using the “Clodo” relation you saw earlier
• Create a set of those classifiers of referrers. That means a set of things like “tuple, dict of someclass”, not of
actual referring objects.
• Group together all the objects with the same set of classifiers of referrers.
So now we know that we have a lot of objects referenced only by one or more dicts (still not very useful) and also a
lot of them referenced by one “normal” dict, referenced by the dict of (meaning “an attribute of”) ebuild_src.package,
and referenced by a WeakRef. Hmm, I wonder what those are. But let’s store this view of the data first, since it took a
while to generate (“_” is a feature of the python interpreter, it’s always the last result):
>>> byrcs = _
>>> byrcs[5]
Partition of a set of 36009 objects. Total size = 1773024 bytes.
Index Count
%
Size
% Cumulative % Referrers by Kind (class / dict of class)
0 36009 100 1773024 100
1773024 100 dict (no owner), dict of
pkgcore.ebuild.ebuild_src.package, weakref.KeyedRef
Erm, yes, we knew that already. If you look in the top right of the table you can see it is still grouping the items by the
kind of their referrer, which is not very useful here. To get more information we can change what they are grouped by:
>>> byrcs[5].byclodo
Partition of a set of 36009 objects. Total size = 1773024 bytes.
Index Count
%
Size
% Cumulative % Kind (class / dict of class)
0 36009 100 1773024 100
1773024 100 str
>>> byrcs[5].bysize
Partition of a set of 36009 objects. Total size = 1773024 bytes.
Index Count
%
Size
% Cumulative % Individual Size
0 10190 28
489120 28
489120 28
48
1
7584 21
394368 22
883488 50
52
2
7335 20
322740 18
1206228 68
44
3
3947 11
221032 12
1427260 80
56
4
3364
9
134560
8
1561820 88
40
5
1903
5
114180
6
1676000 95
60
6
877
2
56128
3
1732128 98
64
7
285
1
19380
1
1751508 99
68
8
451
1
16236
1
1767744 100
36
9
57
0
4104
0
1771848 100
72
This took the set of objects with that odd set of referrers and redisplayed them grouped by “clodo”. So now we know
they’re all strings. Most of them are pretty small too. To get some idea of what we’re dealing with we can pull some
random examples out:
3.1. Content
35
pkgcore Documentation, Release trunk
>>> byrcs[5].byid
Set of 36009 <str> objects. Total size = 1773024 bytes.
Index
Size
%
Cumulative %
Representation (limited)
0
80
0.0
80
0.0 ’media-plugin...re20051219-r1’
1
76
0.0
156
0.0 ’app-emulatio...4.20041102-r1’
2
76
0.0
232
0.0 ’dev-php5/ezc...hemaTiein-1.0’
3
76
0.0
308
0.0 ’games-misc/f...wski-20030120’
4
76
0.0
384
0.0 ’mail-client/...pt-viewer-0.8’
5
76
0.0
460
0.0 ’media-fonts/...-100dpi-1.0.0’
6
76
0.0
536
0.0 ’media-plugin...gdemux-0.10.4’
7
76
0.0
612
0.0 ’media-plugin...3_pre20051219’
8
76
0.0
688
0.0 ’media-plugin...3_pre20051219’
9
76
0.0
764
0.0 ’media-plugin...3_pre20060502’
>>> byrcs[5].byid[0].theone
’media-plugins/vdr-streamdev-server-0.3.3_pre20051219-r1’
A pattern emerges! (sets with one item have a “theone” attribute with the actual item, all sets have a “nodes” attribute
that returns an iterator yielding the items).
We could have used another heapy trick to get a better idea of what kind of string this was:
>>> byrcs[5].byvia
Partition of a set of 36009 objects. Total size = 1773024 bytes.
Index Count
%
Size
% Cumulative % Referred Via:
0
1
0
80
0
80
0 "[’cpvstr’]", ’.key’,
1
1
0
76
0
156
0 "[’cpvstr’]", ’.key’,
2
1
0
76
0
232
0 "[’cpvstr’]", ’.key’,
3
1
0
76
0
308
0 "[’cpvstr’]", ’.key’,
4
1
0
76
0
384
0 "[’cpvstr’]", ’.key’,
5
1
0
76
0
460
0 "[’cpvstr’]", ’.key’,
6
1
0
76
0
536
0 "[’cpvstr’]", ’.key’,
7
1
0
76
0
612
0 "[’cpvstr’]", ’.key’,
8
1
0
76
0
688
0 "[’cpvstr’]", ’.key’,
9
1
0
76
0
764
0 "[’cpvstr’]", ’.key’,
<35999 more rows. Type e.g. ’_.more’ to view.>
’.keys()[23147]’
’.keys()[12285]’
’.keys()[12286]’
’.keys()[16327]’
’.keys()[17754]’
’.keys()[19079]’
’.keys()[21704]’
’.keys()[23473]’
’.keys()[24239]’
’.keys()[3070]’
Ouch, 36009 total rows for 36009 objects. What this did is similar to what “byrcs” did: for every object in the set it
determined how they can be reached through their referrers, then groups objects that can be reached in the same ways
together. Unfortunately it is grouping everything reachable as a dictionary key differently, so this is not very useful.
XXX WTF XXX
It is not likely this accomplishes anything, but let’s assume we want to know if there are any objects in this set not
reachable as the “key” attribute. Heapy can tell us (although this is very slow... there might be a better way but I do
not know it yet):
>>> nonkeys = byrcs[5] & hp.Via(’.key’).alt(’<’)
>>> nonkeys.byrcs
hp.Nothing
(remember “hp” was our main entrance into heapy, the object that gave us the set of all objects we’re interested in
earlier).
What does this do? “hp.Via(‘.key’)” creates a “symbolic set” of “all objects reachable only as the ‘key’ attribute of
something” (it’s a “symbolic set” because there are no actual objects in it). The “alt” method gives us a new symbolic
set of everything reachable via “less than” this way. We then intersect this with our set and discover there is nothing
left.
A similar construct that does not do what we want is:
36
Chapter 3. Developer Notes
pkgcore Documentation, Release trunk
>>> nonkeys = byrcs[5] & ~hp.Via(’.key’)
The “~” operator inverts the symbolic set, giving a set matching everything not reachable exactly as a “key” attribute.
The key word here is “exactly”: since everything in our set was also reachable in two other ways this intersection
matches everything.
Ok, let’s get back to the stuff actually eating memory:
>>> h[0].byrcs
Index Count
%
Size
0 670791 68 31716096
1 139232 14 6525856
2 136558 14 6042408
3 36009
4 1773024
4
5
6
7
8
9
1762
824
140
194
30
55
0
0
0
0
0
0
107772
69476
31312
11504
6284
1972
% Cumulative % Referrers by Kind (class / dict of class)
68 31716096 68 dict (no owner)
14 38241952 83 tuple
13 44284360 96 dict of pkgcore.ebuild.ebuild_src.package
4 46057384 99 dict (no owner), dict of
pkgcore.ebuild.ebuild_src.package, weakref.KeyedRef
0 46165156 100 list
0 46234632 100 types.CodeType
0 46265944 100 function, tuple
0 46277448 100 dict of module
0 46283732 100 dict of type
0 46285704 100 dict of module, tuple
Remember h[0] gave us all str objects, so this is all string objects grouped by the kind(s) of their referrers. Also notice
index 3 here is the same set of stuff we saw earlier:
>>> h[0].byrcs[3] ^ byrcs[5]
hp.Nothing
Most operators do what you would expect, & intersects for example.
“We have a lot of strings in dicts” is not that useful either, let’s see if we can narrow that down a little:
>>> h[0].byrcs[0].referrers.byrcs
Partition of a set of 44124 objects. Total size = 18636768 bytes.
Index Count
%
Size
% Cumulative % Referrers by Kind (class / dict of class)
0 24681 56 12834120 69 12834120 69 dict of pkgcore.ebuild.ebuild_src.package
1 19426 44 5371024 29 18205144 98 dict (no owner)
2
1
0
393352
2 18598496 100 dict of pkgcore.repository.prototype.IterValLazyDict
3
1
0
6280
0 18604776 100 __builtin__.set
4
1
0
6280
0 18611056 100 dict of module, guppy.heapy.heapyc.RootStateType
5
1
0
6280
0 18617336 100 dict of pkgcore.ebuild.eclass_cache.cache
6
1
0
6280
0 18623616 100 dict of
pkgcore.repository.prototype.PackageIterValLazyDict
7
4
0
5536
0 18629152 100 type
8
4
0
3616
0 18632768 100 dict of type
9
1
0
1672
0 18634440 100 dict of module, dict of os._Environ
(Broken down: h[0].byrcs[0] is the set of all str objects referenced only by dicts, h[0].byrcs[0].referrers is the set of
those dicts, and the final .byrcs displays those dicts grouped by their referrers)
Keep an eye on the size column. We have over 12M worth of just dicts (not counting the stuff in them) referenced only
as attribute of ebuild_src.package. If we include the stuff kept alive by those dicts we’re talking about a big chunk of
the 100MB total here:
>>> t = _
>>> t[0].domisize
61269552
60M out of our 100M would be deallocated if we killed those dicts. So let’s ask heapy what dicts that are:
3.1. Content
37
pkgcore Documentation, Release trunk
>>> t[0].byvia
Partition of a set of 24681 objects. Total size = 12834120 bytes.
Index Count
%
Size
% Cumulative % Referred Via:
0 24681 100 12834120 100 12834120 100 "[’data’]"
(it is easy to get confused by the “byrcs” view of our “t”. t[0] is not a bunch of “dict of ebuild_src.package”. It is a
bunch of dicts with strings in them, namely those that are referred to by the dict of ebuild_src.package, and not by
anything else. So the byvia output means those dicts with strings in them are all “data” attributes of ebuild_src.package
instances).
(sidenote: earlier we saw byvia say ”.key”, now it says “[’data’]”. It’s different because the previous type used
__slots__ (so there was no “dict of” involved) and this type does not (so there is a “dict of” and our dicts are the “data”
key in it).
So what is in the dicts:
>>> t[0].referents
Partition of a set of 605577 objects. Total size = 34289392 bytes.
Index Count
%
Size
% Cumulative % Kind (class / dict of class)
0 556215 92 27710068 81 27710068 81 str
1 24681
4 6085704 18 33795772 99 dict (no owner)
2 24681
4
493620
1 34289392 100 long
>>> _.byvia
Partition of a set of 605577 objects. Total size = 34289392 bytes.
Index Count
%
Size
% Cumulative % Referred Via:
0 24681
4 6085704 18
6085704 18 "[’_eclasses_’]"
1 21954
4 3742976 11
9828680 29 "[’DEPEND’]"
2 22511
4 3300052 10 13128732 38 "[’RDEPEND’]"
3 24202
4 2631304
8 15760036 46 "[’SRC_URI’]"
4 24681
4 1831668
5 17591704 51 "[’DESCRIPTION’]"
5 24674
4 1476680
4 19068384 56 "[’HOMEPAGE’]"
6 24681
4 1297680
4 20366064 59 "[’KEYWORDS’]"
7 24681
4
888516
3 21254580 62 ’.keys()[3]’
8 24681
4
888516
3 22143096 65 ’.keys()[9]’
9 24681
4
810108
2 22953204 67 "[’LICENSE’]"
<32 more rows. Type e.g. ’_.more’ to view.>
Strings, nested dicts and longs, and most size eaten up by the “_eclasses_” values. There is also a significant amount
eaten up by keys values, which is a bit odd, so let’s investigate:
>>> refs = t[0].referents
>>> i=iter(refs.byvia[7].nodes)
>>> i.next()
’DESCRIPTION’
>>> i.next()
’DESCRIPTION’
>>> i.next()
’DESCRIPTION’
>>> i.next()
’DESCRIPTION’
>>> i.next()
’DESCRIPTION’
Eep!
>>> refs.byvia[7].bysize
Partition of a set of 24681 objects. Total size = 888516 bytes.
Index Count
%
Size
% Cumulative % Individual Size
0 24681 100
888516 100
888516 100
36
38
Chapter 3. Developer Notes
pkgcore Documentation, Release trunk
It looks like we have 24681 identical strings here, using up about 1M of memory. The other odd entry is the
‘_eclasses_’ string apparently.
Extra stuff for c extension developers
To provide accurate statistics if your code uses extension types you must provide heapy with a way to get the following
data for your custom types:
• How large is a certain instance?
• What objects does an instance contain?
• How does the instance refer to a contained object?
You provide these through a NyHeapDef struct, defined in heapdef.h in the guppy source. This header is not installed,
so you should just copy it into your source tree. It is a good idea to read this header file side by side with the following
descriptions, since it contains details omitted here. The stdtypes.c file contains implementations for the basic python
types which you can read for inspiration.
The NyHeapDef struct provides heapy with three function pointers:
SizeGetter
To answer “how large is an instance” you provide a NyHeapDef_SizeGetter function that is called with a PyObject*
and returns an int: the number of bytes the object occupies. If you do not provide this function heapy uses a default that
looks at the tp_basicsize and tp_itemsize fields of the type. This means that if you do not allocate any extra memory
for non-python objects (e.g. for c strings) you do not need to provide this function.
Traverser
To answer “What objects does an instance contain” you provide a traversal function (NyHeapDef_Traverser). This is
called with a pointer to a “visit procedure”, an instance of your extension type and some other stuff. You should then
call the visit procedure for every python object contained in your object.
This might sound familiar: to support the python garbage collector you provide a very similar function (tp_traverse).
Actually heapy will use tp_traverse if you do not provide a heapy-specific traverse function. Doing this makes sense
if you do not support the garbage collector for some reason, or if you contain objects that are irrelevant to the garbage
collector.
An example would be a type that contains a single python string object (that no other code can get a reference to). If
this object does not have references to other python objects it cannot be involved in cycles so supporting gc would be
useless. However you do still want heapy to know about the memory occupied by the contained string. You could do
that by adding that size in your NyHeapDef_SizeGetter function but it is probably easier to tell heapy about the string
through the traversal function (so you do not have to calculate the memory occupied by the string).
If the above type would also have a reference to some arbitrary (non-private) python object it should support gc, but
it does not need to tell gc about the contained string. So you would have two traversal functions, one for heapy that
visits the string and one for gc that does not.
RelationGetter
The last function heapy wants tells it in what way your instance refers to some contained object. It is used to provide
the “byvia” view. This calls a visit function once for each way your instance refers to a target object, telling it what
kind of reference it is.
3.1. Content
39
pkgcore Documentation, Release trunk
Providing the heapdef struct to heapy
Once you have the needed function pointers in a struct you need to pass this to heapy somehow. This is done through
a standard cpython mechanism called “cobjects”. From python these look like rather stupid objects you cannot do
anything with, but from c you can pull out a void* that was put in when the object was constructed. You can wrap an
arbitrary pointer in a CObject, make it available as attribute of your module, then import it from some other module,
pull the void* back out and cast it to the original type.
heapy looks for a _NyHeapDefs_ attribute on all loaded modules. If this attribute exists and is a CObject the pointer in
it is used as a pointer to an array of NyHeapDef struct (terminated with a struct with only nulls). Example code doing
this is in sets.c in the guppy source.
3.1.16 Plugins system
Goals
The plugin system (pkgcore.plugin) is used to pick up extra code (potentially distributed separately from pkgcore
itself) at a place where using the config system is not a good idea for some reason. This means that for a lot of things
that most people would call “plugins” you should not actually use pkgcore.plugin, you should use the config
system. Things like extra repository types should simply be used as “class” value in the configuration. The plugin
system is currently mainly used in places where handing in a ConfigManager is too inconvenient.
Using plugins
Plugins are looked up based on a string “key”. You can always look up all available plugins matching this key with
pkgcore.plugin.get_plugins(key). For some kinds of plugin (the ones defining a “priority” attribute) you
can also get the “best” plugin with pkgcore.plugin.get_plugin(key). This does not make sense for all
kinds of plugin, so not all of them define this.
The plugin system does not care about what kind of object plugins are, this depends entirely on the key.
Adding plugins
Basics, caching
Plugins for pkgcore are loaded from modules inside the pkgcore.plugins package. This package has some magic
to make plugins in any subdirectory pkgcore/plugins under a directory on sys.path work. So if pkgcore
itself is installed in site-packages you can still add plugins to /home/you/pythonlib/pkgcore/plugins if
/home/you/pythonlib is in PYTHONPATH. You should not put an __init__.py in this extra plugin directory.
Plugin modules should contain a pkgcore_plugins directory that maps the “key” strings to a sequence of plugins.
This dictionary has to be constant, since pkgcore keeps track of what plugin module provides plugins for what keys in
a cache file to avoid unnecessary imports. So this is invalid:
try:
import spork_package
except ImportError:
pkgcore_plugins = {}
else:
pkgcore_plugins = {’myplug’: [spork_package.ThePlugin]}
since if the plugin cache is generated while the package is not available pkgcore will cache the module as not providing
any myplug plugins, and the cache will not be updated if the package becomes available (only changes to the mtime
of actual plugin modules invalidate the cache). Instead you should do something like this:
40
Chapter 3. Developer Notes
pkgcore Documentation, Release trunk
try:
from spork_package import ThePlugin
except ImportError:
class ThePlugin(object):
disabled = True
pkgcore_plugins = {’myplug’: [ThePlugin]}
If a plugin has a “disabled” attribute the plugin system will never return it from get_plugin or get_plugins.
Priority
If you want your plugin to support get_plugin it should have a priority attribute: an integer indicating how
“preferred” this plugin is. The plugin with the highest priority (that is not disabled) is returned from get_plugin.
Some types of plugins need more information to determine a priority value. Those should not have a priority attribute.
They should use get_plugins instead and have a method that gets passed the extra data and returns the priority.
Import behaviour
Assuming the cache is working correctly (it was generated after installing a plugin as root) pkgcore will import all
plugin modules containing plugins for a requested key in priority order until it hits one that is not disabled. The
“disabled” value is not cached (a plugin that is unconditionally disabled makes no sense), but the priority value is. You
can fake a dynamic priority by having two instances of your plugin registered and only one of them enabled at the
same time.
This means it makes sense to have only one kind of plugin per plugin module (unless the required imports overlap):
this avoids pulling in imports for other kinds of plugin when one kind of plugin is requested.
The disabled value is not cached by the plugin system after the plugin module is imported. This means it should be a
simple attribute (either completely constant or set at import time) or property that does its own caching.
Adding a plugin package
Both get_plugin and get_plugins take a plugin package as second argument. This means you can use the
plugin system for external pkgcore-related tools without cluttering up the main pkgcore plugin directory. If you
do this you will probably want to copy the __path__ trick from pkgcore/plugin/__init__.py to support
plugins elsewhere on sys.path.
3.1.17 Pkgcore/Portage differences
Disclaimer
Pkgcore moves fairly fast in terms of development- we will strive to keep this doc up to date, but it may lag behind the
actual code.
Ebuild environment changes
All changes are either glep33 related, or a tightening of the restrictions on the env to block common snafus that localize
the ebuilds environment to that machine.
3.1. Content
41
pkgcore Documentation, Release trunk
• portageq based functions are disabled in the global scope.
Reasoning for this is that of QAhas_version/best_version must not affect the generated metadata. As such, portageq calls in the global scope
are disabled.
• inherit is disabled in all phases but depend and setup. Folks no longer do it, but inherit from within one of the
build/install phases is now actively blocked.
• The ebuild env is now effectively akin to suspending the process, and restarting it. Essentially, transitioning
between ebuild phases, the ebuild environment is snapshotted, cleaned of irrevelent data (bash forced vars for
example, or vars that pkgcore sets for the local system on each shift into a phase), and saved. Portage does
this partially (re-execs ebuilds/eclasses, thus stomping the env on each phase change), pkgcore does it fully. As
such, pkgcore is capable of glep33, while portage is not (env fixes are the basis of glep33).
• ebuild.sh now protects itself from basic fiddling. Ebuild generated state must work as long as the EAPI is
the same, regardless of the generating portage version, and the portage version that later uses the saved state
(simple example, generated with portage-2.51, if portage 3 is EAPI compliant with that env, it must not allow
it’s internal bash changes to break the env). As such, certain funcs are not modifiable by the ebuild- namely,
internal portage/pkgcore functionality, hasq/useq for example. Those functions that are read-only also are not
saved in the ebuild env (they should be supplied by the portage/pkgcore instance reloading the env).
• ebuild.sh is daemonized. The upshot of this is that regen is roughly 2x faster (careful reuse of ebuild.sh instances
rather then forcing bash to spawn all over). Additional upshot of this is that their are bidirectional communication
pipes between ebuild.sh and the python parent- env inspection, logging, passing requests up to the python side
(has_version/best_version for example) are now handled within the existing processes. Design of it from the
python side is that of an extensible event handler, as such it’s extremely easy to add new commands in, or special
case certain things.
Repository Enhancements
Pkgcore internally uses a sane/uniform repository abstraction- the benefits of this are:
• repository class (which implements the accessing of the on disk/remote tree) is pluggable. Remote vdb/portdir
is doable, as is having your repository tree ran strictly from downloaded metadata (for example), or running
from a tree stored in a tarball/zip file (mildly crazy, but it’s doable).
• seperated repository instances. We’ve not thrown out overlays (as paludis did), but pkgcore doesn’t force every
new repository to be an overlay of the ‘master’ PORTDIR as portage does.
• optimized repository classes- for the usual vdb and ebuild repository (those being on disk backwards compatible
with portage 2.x), the number of syscalls required was drastically reduced, with ondisk info (what packages
available per category for example) cached. It is a space vs time trade off, but the space trade off is neglible
(couple of dict with worst case, 66k mappings)- as is, portage’s listdir caching consumed a bit more memory
and was slower, so all in all a gain (mainly it’s faster with using slightly less memory then portages caching).
• unique package instances yielded from repository. Pkgcore uses a package abstraction internally for accessing
metadata/version/category, etc- all instances returned from repositories are unique immutable instances. Gain
of it is that if you’ve got dev-util/diffball-0.7.1 sitting in memory already, it will return that instance instead of
generating a new one- and since metadata is accessed via the instance, you get at most one load from the cache
backend per instance in memory- cache pull only occurs when required also. As such, far faster for when doing
random package accessing and storing of said packages (think repoman, dependency resolution, etc).
3.1.18 Tackling domain
tag a ‘x’ in front of stuff that’s been implemented
unhandled (eg, figure these out) vars/features
42
Chapter 3. Developer Notes
pkgcore Documentation, Release trunk
• (user)?sandbox
• userpriv(_fakeroot)?
• digest
• cvs (this option is a hack)
• fixpackages , which probably should be a sync thing (would need to bind the vdb and binpkg repo to it though)
• keep(temp|work), easy to implement, but where to define it?
• PORT_LOGDIR
• env overrides of use...
vdb wrapper/vdb repo instantiation (either domain created wrapper, or required in the vdb repo section def)
• CONFIG_PROTECT*
• collision-protect
• no(doc|man|info|clean) (wrapper/mangler)
• suidctl
• nostrip. in effect, strip defaults to on; wrappers if after occasionally on, occasionally off.
• sfperms
build section (vars)
• C(HOST|TARGET), (LD*|C*)FLAGS?
• (RESUME|FETCH)COMMAND are fetcher things, define it there.
• MAKEOPTS
• PORTAGE_NICENESS (imo)
• TMPDIR ? or domain it?
gpg is bound to repo, class type specifically. strict/severe are likely settings of it. the same applies for profiles.
distlocks is a fetcher thing, specifically (probably) class type.
buildpkgs is binpkg + filters.
package.provided is used to generate a seperate vdb, a null vdb that returns those packages as installed.
3.1.19 Testing
We use twisted.trial for our tests, to run the test framework run:
trial pkgcore
Your own tests must be stored in pkgcore.test - furthermore, tests must pass when ran repeatedly (-u option). You will
want at least twisted-2.2 for that, <2.2 has a few false positives.
Testing for negative assertions
When coding it’s easy to write test cases asserting that you get result xyz from foo, usually asserting the correct flow.
This is ok if nothing goes wrong, but that doesn’t normally happen. :)
3.1. Content
43
pkgcore Documentation, Release trunk
Negative assertions (there probably is a better term for it) means asserting failure conditions and ensuring that the code
handles zyx properly when it gets thrown at it. Most test cases seem to miss this, resulting in bugs being able to hide
away for when things go wrong.
Using –coverage
When writing tests for your code (or for existing code without any tests), it is very useful to use –coverage. Run
trial –coverage <path/to/test>, and then check <cwd>/_trial_temp/coverage/<test/module/name>. Any lines prefixed
with ‘>>>>>’ have not been covered by your tests. This should be rectified before your code is merged to mainline
(though this is not always possible). Those lines prefixed with a number show the number of times that line of code is
evaluated.
3.1.20 perl CPAN
• makeCPANstub in Gentoo/CPAN.pm , dumps cpan config
• screen scraping to get deps, example page http://kobesearch.cpan.org/, use getCPANInfo from CPAN
• use FindDeps for this
• use unmemoize(func) to back out the memoizing of a func; do this on FindDeps
3.1.21 dpkg
this is just basic notes, nothing more. If you know details, fill in the gaps kindly
repos are combined.
Sources.gz (list of source based deb’s) holds name, version, and build deps.
Packages.gz (binary debs, dpkgs) name, version, size, short and long description, bindeps.
repository layout:
dists
stable
main
arch #binary-arm fex
source #?
contrib #?
arch # binary-arm fex
source
non-free # guess.
arch
source
testing...
unstable...
arch/binary-* dirs hold Packages.gz, and Release (potentially) source dirs hold Sources.gz and Release (optionally)
has preinst, postinst, prerm, postrm Same semantics as ebuilds in terms of when to run (coincidence? :)
in dpkg
Build-Depends
Depends
Pre-Depends
Conflicts
44
in ebuild
our DEPEND
our RDEPEND
configure time DEPEND
blockers, affected by Essential (read up on this in debian policy guide)
Chapter 3. Developer Notes
pkgcore Documentation, Release trunk
3.1.22 WARNING
This is the original brain dump from harring; it is not guranteed to be accurate to the current design, it’s kept around
to give an idea of where things came from to contrast to what is in place now.
3.1.23 Introduction
e’yo. General description of layout/goals/info/etc, and semi sortta api.
That and aggregator of random ass crazy quotes should people get bored.
DISCLAIMER
This ain’t the code.
In other words, the actual design/code may be radically different, and this document probably will trail any major
overhauls of the design/code (speaking from past experience).
Updates welcome, as are suggestions and questions- please dig through all documentations in the dir this doc is in
however, since there is a lot of info (both current and historical) related to it. Collapsing info into this doc is attempted,
but explanation of the full restriction protocol (fex) is a lot of info, and original idea is from previous redesign err...
designs. Short version, historical, but still relevant info for restriction is in layout.txt. Other subsystems/design choices
have their basis quite likely from other docs in this directory, so do your homework please :)
Terminology
cp category/package
cpv category/package-version
ROOT livefs merge point, fex /home/bharring/embedded/arm-target or more commonly, root=/
vdb /var/db/pkg, installed packages database.
domain combination of repositories, root, and build information (use flags, cflags, etc). config data + repositories
effectively.
repository trees. ebuild tree (/usr/portage), binpkg tree, vdb tree, etc.
protocol python name for design/api. iter() fex, is a protocol; for iter(o) it does i=o.__iter__(); the returned object is
expected to yield an element when i.next() is called, till it runs out of elements (then throwing a StopIteration).
hesitate to call it defined hook on a class/instance, but this (crappy) description should suffice.
seq sequence, lists/tuples
set list without order (think dict.keys())
General design/idea/approach/requirements
All pythonic components installed by pkgcore must be within pkgcore.* namespace. No more polluting python’s
namespace, plain and simple. Third party plugins to pkgcore aren’t bound by this however (their mess, not ours).
API flows from the config definitions, everything internal is effectively the same. Basically, config data gives you your
starter objects which from there, you dig deeper into the innards as needed action wise.
The general design is intended to heavily abuse OOP. Further, delegation of actions down to components must be
abided by, example being repo + cache interaction. repo does what it can, but for searching the cache, let the cache
do it. Assume what you’re delegating to knows the best way to handle the request, and probably can do it’s job better
then some external caller (essentially).
3.1. Content
45
pkgcore Documentation, Release trunk
Actual configuration is pretty heavily redesigned. Classes and functions that should be constructed based on data from
the user’s configuration have a “hint” describing their arguments. The global config class uses these hints to convert
and typecheck the values in the user’s configuration. Actual configuration file reading and type conversion is done by
a separate class, meaning the global manager is not tied to a single format, or even to configuration read from a file on
disk.
Encapsulation, extensibility/modularity, delegation, and allowing parallelizing of development should be key focuses
in implementing/refining this high level design doc. Realize parallelizing is a funky statement, but it’s apt; work on
the repo implementations can proceed without being held up by cache work, and vice versa.
Final comment re: design goals, defining chunks of callable code and plugging it into the framework is another bit of
a goal. Think twisted, just not quite as prevalent (their needs/focus is much different from ours, twisted is the app,
your code is the lib, vice versa for pkgcore).
Back to config. Here’s general notion of config ‘chunks’ of the subsystem, (these map out to run time objects unless
otherwise stated):
domain
+-- profile (optional)
+-- fetcher (default)
+-- repositories
+-- resolver (default)
+-- build env data?
|
never actually instantiated, no object)
\-- livefs_repo (merge target, non optional)
repository
+-- cache
(optional)
+-- fetcher (optional)
+-- sync
(optional, may change)
\-- sync cache (optional, may chance)
profile
+-- build env?
+-- sets (system mainly).
\-- visibility wrappers
domain is configuration data, accept_(license|keywords), use, cflags, chost, features, etc. profile, dependent on the
profile class you choose is either bound to a repository, or to user defined location on disk (/etc/portage/profile fex).
Domain knows to do incremental crap upon profile settings, lifting package.* crap for visibility wrappers for repositories also.
repositories is pretty straightforward. portdir, binpkg, vdb, etc.
Back to domain. Domain’s are your definition of pretty much what can be done. Can’t do jack without a domain,
period. Can have multiple domains also, and domains do not have to be local (remote domains being a different class
type). Clarifying, think of 500 desktop boxes, and a master box that’s responsible for managing them. Define an appropriate domain class, and appropriate repository classes, and have a config that holds the 500 domains (representing
each box), and you can push updates out via standard api trickery. In other words, the magic is hidden away, just define
remote classes that match defined class rules (preferably inheriting from the base class, since isinstance sanity checks
will become the norm), and you could do emerge –domain some-remote-domain -u glsa on the master box. Emerge
won’t know it’s doing remote crap. Pkgcore won’t even. It’ll just load what you define in the config.
Ambitious? Yeah, a bit. Thing to note, the remote class additions will exist outside of pkgcore proper most likely.
Develop the code needed in parallel to fleshing pkgcore proper out.
Meanwhile, the remote bit + multiple domains + class overrides in config definition is _explicitly_ for the reasons
above. That and x-compile/embedded target building, which is a bit funkier.
Currently, portage has DEPEND and RDEPEND. How do you know what needs be native to that box to build the
46
Chapter 3. Developer Notes
pkgcore Documentation, Release trunk
package, what must be chost atoms? Literally, how do you know which atoms, say the toolchain, must be native vs
what package’s headers/libs must exist to build it? We need an additional metadata key, BDEPEND (build depends).
If you have BDEPEND, you know what actually is ran locally in building a package, vs what headers/libs are required.
Subtle difference, but BDEPEND would allow (with a sophisticated depresolver) toolchain to be represented in deps,
rather then the current unstated dep approach profiles allow.
Aside from that, BDEPEND could be used for x-compile via inter-domain deps; a ppc target domain on a x86 box
would require BDEPEND from the default domain (x86). So... that’s useful.
So far, no one has shot this down, moreso, come up with reasons as to why it wouldn’t work, the consensus thus far is
mainly “err, don’t want to add the deps, too much work”. Regarding work, use indirection.
virtual/toolchain-c metapkg (glep37) that expands out (dependent on arch) into whatever is required to do building
of c sources
virtual/toolchain-c++ same thing, just c++
virtual/autootols take a guess.
virtual/libc this should be tagged into rdepends where applicable, packages that directly require it (compiled crap
mainly)
Yes it’s extra work, but the metapkgs above should cover a large chunk of the tree, say >90%.
Config design
Portage thus far (<=2.0.51*) has had variable ROOT (livefs merge point), but no way to vary configuration data aside
from via a buttload of env vars. Further, there has been only one repository allowed (overlays are just that, extensions
of the ‘master’ repository). Addition of support of any new format is mildly insane due to hardcoding up the wing
wang in the code, and extension/modification of existing formats (ebuild) has some issues (namely the doebuild block
of code).
Goal is to address all of this crap. Format agnosticism at the repository level is via an abstracted repository design that
should supply generic inspection attributes to match other formats. Specialized searching is possible via match, thus
extending the extensibility of the prototype repository design.
Format agnosticism for building/merging is somewhat reliant on the repo, namely package abstraction, and abstraction
of building/merging operations.
On disk configurations for alternatives formats is extensible via changing section types, and plugging them into the
domain definition.
Note alt. formats quite likely will never be implemented in pkgcore proper, that’s kind of the domain of pkgcore
addons. In other words, dpkg/rpm/whatever quite likely won’t be worked on by pkgcore developers, at least not in the
near future (too many other things to do).
The intention is to generalize the framework so it’s possible for others to do so if they choose however.
Why is this good? Ebuild format has issues, as does our profile implementation. At some point, alternative
formats/non-backwards compatible tweaks to the formats (ebuild or profile) will occur, and then people will be quite
happy that the framework is generalized (seriously, nothing is lost from a proper abstracted design, and flexibility/power is gained).
config’s actions/operation
pkgcore.config.load_config() is the entrance point, returns to you a config object (pkgcore.config.central). This object
gives you access to the user defined configs, although only interest/poking at it should be to get a domain object from
it.
3.1. Content
47
pkgcore Documentation, Release trunk
domain object is instantiated by config object via user defined configuration. domains hold instantiated repositories,
bind profile + user prefs (use/accept_keywords) together, and _should_ simplify this data into somewhat user friendly
methods. (define this better).
Normal/default domain doesn’t know about other domains, nor give a damn. Embedded targets are domains, and
_will_ need to know about the livefs domain (root=/), so buildplan creation/handling may need to be bound into
domains.
Objects/subsystems/stuff
So... this is general naming of pretty much top level view of things, stuff emerge would be interested in (and would
fool with). hesitate to call it a general api, but it probably will be as such, exempting any abstraction layer/api over all
of this (good luck on that one }:] ).
IndexableSequence
functions as a set and dict, with caching and on the fly querying of info. mentioned due to use in repository and other
places... (it’s a useful lil sucker)
This actually is misnamed. the order of iteration isn’t necessarily reproducable, although it’s usually constant. IOW,
it’s normally a sequence, but the class doesn’t implicitly force it
LazyValDict
similar to ixseq, late loading of keys, on fly pulling of values as requested.
global config object (from pkgcore.config.load_config())
see config.rst.
domain object
bit of debate on this one I expect. any package.{mask,unmask,keywords} mangling is instantiated as a wrapper around
repository instances upon domain instantiation. code should be smart and lift any package.{mask,unmask,keywords}
wrappers from repositoriy instances and collapse it, pointing at the raw repo (basically don’t have N wrappers, collapse
it into a single wrapper). Not worth implementing until the wrapper is a faster implementation then the current
pkgcore.repository.visibility hack though (currently O(N) for each pkg instance, N being visibility restrictions/atoms).
Once it’s O(1), collapsing makes a bit more sense (can be done in parallel however).
a word on inter repository dependencies... simply put, if the repository only allows satisfying deps from the same
repository, the package instance’s *DEPEND atom conversions should include that restriction. Same trickery for
keeping ebuilds from depping on rpm/dpkg (and vice versa).
.repositories in the air somewhat on this one. either indexablesequence, or a repositorySet. Nice aspect of the latter
is you can just use .match with appropriate restrictions. very simply interface imo, although should provide
a way to pull individual repositories/labels of said repos from the set though. basically, mangle a .raw_repo
indexablesequence type trick (hackish, but nail it down when reach that bridge)
build plan creation
<TODO insert details as they’re fleshed out>
48
Chapter 3. Developer Notes
pkgcore Documentation, Release trunk
sets
TODO chuck in some details here. probably defined via user config and/or profile, although what’s it define?
atoms/restrictions? itermatch might be useful for a true set.
build/setup operation
(need a good name for this; dpkg/rpm/binpkg/ebuild’s ‘prepping’ for livefs merge should all fall under this, with
varying use of the hooks)
.build() do everything, calling all steps as needed
.setup() whatever tmp dirs required, create ‘em.
.req_files() (fetchables, although not necessarily with url (restrict=”fetch”...)
.unpack() guess.
.configure() unused till ebuild format version two (ya know, that overhaul we’ve been kicking around? :)
.compile() guess.
.test() guess.
.install() install to tmp location. may not be used dependent on the format.
.finalize() good to go. generate (jit?) contents/metadata attributes, or returns a finalized instance should generate a
immutable package instance.
repo change operation
base class.
.package package instance of what the action is centering around.
.start() notify repo we’re starting (locking mainly, although prerm/preinst hook also)
.finish() notify repo we’re done.
.run() high level, calls whatever funcs needed. individual methods are mainly for ui, this is if you don’t display “doing
install now... done... doing remove now... done” stuff.
remove operation
derivative of repo change operation.
.remove() guess.
.package package instance of what’s being yanked.
install operation
derivative of repo change operation
.package what’s being installed.
.install() install it baby.
3.1. Content
49
pkgcore Documentation, Release trunk
merge operation
derivative of repo remove and install (so it has .remove and .install, which must be called in .install and .remove order)
.replacing package instance of what’s being replaced.
.package what’s being installed
fetchables
basically a dict of stuff jammed together, just via attribute access (think c struct equiv)
.filename
.url tuple/list of url’s.
.chksums dict of chksum:val
fetcher
hey hey. take a guess.
worth noting, if fetchable lacks .chksums["size"], it’ll wipe any existing file. if size exists, and existing file is
bigger, wipe file, and start anew, otherwise resume. mirror expansion occurs here, also.
.fetch(fetchable, verifier=None) # if verifier handed in, does verification.
verifier
note this is basically lifted conceptually from mirror_dist. if wondering about the need/use of it, look at that source.
verify() handed a fetchable, either False or True
repository
this should be format agnostic, and hide any remote bits of it. this is general info for using it, not designing a repository
class
.mergable() true/false. pass a pkg to it, and it reports whether it can merge that or not.
.livefs boolean, indicative of whether or not it’s a livefs target- this is useful for resolver, shop it to other repos, binpkg
fex prior to shopping it to the vdb for merging to the fs. Or merge to livefs, then binpkg it while continuing
further building dependent on that package (ui app’s choice really).
.raw_repo either it weakref’s self, or non-weakref refs another repo. why is this useful? visibility wrappers... this
gives ya a way to see if p.mask is blocking usable packages fex. useful for the UI, not too much for pkgcore
innards.
.frozen boolean. basically, does it account for things changing without it’s knowledge, or does it not. frozen=True is
faster for ebuild trees for example, single check for cache staleness. frozen=False is slower, and is what portage
does now (meaning every lookup of a package, and instantiation of a package instance requires mtime checks
for staleness).
.categories IndexableSequence, if iterated over, gives ya all categories, if getitem lookup, sub-category category
lookups. think media/video/mplayer
50
Chapter 3. Developer Notes
pkgcore Documentation, Release trunk
.packages IndexableSequence, if iterated over, all package names. if getitem (with category as key), packages of that
category.
.versions IndexableSequence, if iterated over, all cpvs. if getitem (with cat/pkg as key), versions for that cp
.itermatch() iterable, given an atom/restriction, yields matching package instances.
.match() def match(self, atom):
return list(self.itermatch(atom)) voila.
.__iter__() in other words, repository is iterable. yields package instances.
.sync() sync, if the repo swings that way. flesh it out a bit, possibly handing in/back ui object for getting updates...
digressing for a moment...
note you can group repositories together, think portdir + portdir_overlay1 + portdir_overlay2. Creation of a repositoryset basically would involve passing multiple instantiating repo’s, and depending on that classes semantics, it internally
handles the stacking (right most positional arg repo overrides 2nd right most, ... overriding left most) So... stating it
again/clearly if it ain’t obvious, everything is configuration/instantiating of objects, chucked around/mangled by the
pkgcore framework.
What isn’t obvious is that since a repository set gets handed instantiated repositories, each repo, including the set
instance, can should be able to have it’s own cache (this is assuming it’s ebuild repos through and through). Why?
Cache data doesn’t change for the most part exempting which repo a cpv is from, and the eclass stacking. Handled
individually, a cache bound to portdir should be valid for portdir alone, it shouldn’t carry data that is a result of eclass
stacking from another overlay + that portdir. That’s the business of the repositoryset. Consequence of this is that the
repositoryset needs to basically reach down into the repository it’s wrapping, get the pkg data, then rerequest the keys
from that ebuild with a different eclass stack. This would be a bit expensive, although once inherit is converted to a
pythonic implementation (basically handing the path to the requested eclass down the pipes to ebuild*.sh), it should
be possible to trigger a fork in the inherit, and note python side that multiple sets of metadata are going to be coming
down the pipe. That should alleviate the cost a bit, but it also makes multiple levels of cache reflecting each repository
instance a bit nastier to pull off till it’s implemented.
So... short version. Harring is a perfectionist, and says it should be this way. reality of the situation makes it a bit
trickier. Anyone interested in attempting the mod, feel free, otherwise harring will take a crack at it since he’s being
anal about having it work in such a fashion.
Or... could do thus. repo + cache as a layer, wrapped with a ‘regen’ layer that handles cache regeneration as required.
Via that, would give the repositoryset a way to override and use it’s own specialized class that ensures each repo gets
what’s proper for it’s layer. Think raw_repo type trick.
continuing on...
cache
ebuild centric, although who knows (binpkg cache ain’t insane ya know). short version, it’s functionally a dict, with
sequence properties (iterating over all keys).
.keys() return every cpv/package in the db.
.readonly boolean. Is it modifiable?
.match() Flesh this out. Either handed a metadata restriction (or set of ‘em), or handed dict with equiv info (like the
former). ebuild caches most likely should return mtime information alongside, although maybe dependent on
readonly. purpose of this? Gives you a way to hand off metadata searching to the cache db, rather then the repo
having to resort to pulling each cpv from the cache and doing the check itself. This is what will make rdbms
cache backends finally stop sucking and seriously rocking, properly implemented at least. :) clarification, you
don’t call this directly, repo.match delegates off to this for metadata only restrictions
3.1. Content
51
pkgcore Documentation, Release trunk
package
this is a wrapped, constant package. configured ebuild src, binpkg, vdb pkg, etc. ebuild repositories don’t exactly and
return this- they return unconfigured pkgs, which I’m not going to go into right now (domains only see this protocol,
visibility wrappers see different)
.depends usual meaning. ctarget depends
.rdepends usual meaning. ctarget run time depends. seq,
.bdepends see ml discussion. chost depends, what’s executed in building this (toolchain fex). seq.
.files get a better name for this. doesn’t encompas files/*, but could be slipped in that way for remote. encompasses restrict fetch (files with urls), and chksum data. seq.
.description usual meaning, although remember probably need a way to merge metadata.xml lond desc into the more
mundane description key.
.license usual meaning, depset
.homepage usual. Needed?
.setup() Name sucks. gets ya the setup operation, which does building/whatever.
.data Raw data. may not exist, don’t screw with it unless you know what it is, and know the instance’s .data layout.
.build() if this package is buildable, return a build operation, else return None
restriction
see layout.txt for more fleshed out examples of the idea. note, match and pmatch have been reversed namewise.
.match() handed package instance, will return bool of whether or not this restriction matches.
.cmatch() try to force the changes; this is dependent on the package being configurable.
.itermatch() new one, debatable. short version, giving a sequence of package instances, yields true/false for them.
why might this be desirable? if setup of matching is expensive, this gives you a way to amoritize the cost. might
have use for glsa set target. define a restriction that limits to installed pkgs, yay/nay if update is avail...
restrictionSet
mentioning it merely cause it’s a grouping (boolean and/or) of individual restrictions an atom, which is in reality a
category restriction, package restriction, and/or version restriction is a boolean and set of restrictions
ContentsRestriction
whats this you say? a restriction for searching the vdb’s contents db? Perish the thought! ;)
metadataRestriction
Mentioning this for the sake of pointing out a subclass of it, DescriptionRestriction- this will be a class representing
matching against description data. See repo.match and cache.match above. The short version is that it encapsulates
the description search (a very slow search right now) so that repo.match can hand off to the cache (delegation), and the
cache can do the search itself, however it sees fit.
52
Chapter 3. Developer Notes
pkgcore Documentation, Release trunk
So... for the default cache, flat_list (19500 ebuilds == 19500 files to read for a full searchDesc), still is slow unless
flat_list gets some desc. cache added to it internally. If it’s a sql based cache, the sql_template should translate the
query into the appropriate select statement, which should make it much faster.
Restating that, delegation is absolutely required. There have been requests to add intermediate caches to the tree, or
move data (whether collapsing metadata.xml or moving data out of ebuilds) so that the form it is stored is in quicker
to search. These approaches are wrong. Should be clear from above that a repository can, and likely will be remote on
some boxes. Such a shift of metadata does nothing but make repository implementations that harder, and shift power
away from what knows best how to use it. Delegation is a massively more powerful approach, allowing for more
extensibility, flexibility and speed.
Final restating- searchDesc is matching against cache data. The cache (whether flat_list, anydbm, sqlite, or a remote
sql based cache) is the authority about the fastest way to do searches of it’s data. Programmers get pist off when users
try and tell them how something internally should be implemented- it’s fundamentally the same scenario. The cache
class the user chooses knows how to do it’s job the best, provide methods of handing control down to it, and let it
do it’s job (delegation). Otherwise you’ve got a backseat driver situation, which doesn’t let those in the know, do the
deciding (cache knows, repo doesn’t).
Mind you not trying to be harsh here. If in reading through the full doc you disagree, question it; if after speeding
up current cache implementation, note that any such change must be backwards compatible, and not screw up the
possibilities of encapsulation/delegation this design aims for.
logging
flesh this out (define this basically). short version, no more writemsg type trickery, use a proper logging framework.
ebuild-daemon.sh
Hardcoded paths have to go. /usr/lib/portage/bin == kill it. Upon initial loadup of ebuild.sh, dump the default/base
path down to the daemon, including a setting for /usr/lib/portage/bin . Likely declare -xr it, then load the actual
ebuild*.sh libs. Backwards compatibility for that is thus, ebuild.sh defines the var itself in global scope if it’s undefined.
Semblence of backwards compatibility (which is actually somewhat pointless since I’m about to blow it out of the
water).
Ebuild-daemon.sh needs a function for dumping a _large_ amount of data into bash, more then just a line or two.
For the ultra paranoid, we load up eclasses, ebuilds, profile.bashrc’s into python side, pipe that to gpg for verification,
then pipe that data straight into bash. No race condition possible for files used/transferred in this manner.
A thought. The screw around speed up hack preload_eclasses added in ebd’s heyday of making it as fast as possible
would be one route; Basically, after verification of an elib/eclass, preload the eclass into a func in the bash env. and
declare -r the func after the fork. This protects the func from being screwed with, and gives a way to (at least per ebd
instance) cache the verified bash code in memory.
It could work surprisingly enough (the preload_eclass command already works), and probably be fairly fast versus the
alternative. So... the race condition probably can be flat out killed off without massive issues. Still leaves a race for
perms on any files/*, but neh. A) That stuff shouldn’t be executed, B) security is good, but we can’t cover every
possibility (we can try, but dimishing returns)
A lesser, but still tough version of this is to use the indirection for actual sourcing to get paths instead. No
EBUILD_PATH, query python side for the path, which returns either ‘’ (which ebd interprets as “err, something
is whacked, time to scream”), or the actual path.
In terms of timing, gpg verification of ebuilds probably should occur prior to even spawning ebd.sh. profile, eclass,
and elib sourcing should use this technique to do on the fly verification though. Object interaction for that one is going
to be really fun, as will be mapping config settings to instantiation of objs.
3.1. Content
53
pkgcore Documentation, Release trunk
54
Chapter 3. Developer Notes
CHAPTER 4
Indices and tables
• genindex
• modindex
• search
55