snakeoil Documentation
Release -trunk
Brian Harring
October 27, 2014
Contents
1
Community
3
2
Getting the source (downloading releases or trunk)
5
3
Snakeoil intentions
3.1 Python annoyances and hard issues . . . . . . . . . . . . . . . . . . . . . . .
3.2 Supporting multiple python versions can be a pain . . . . . . . . . . . . . . .
3.3 Optimizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4 Avoiding Boilerplate (functionality to help with DRY- Don’t Repeat Yourself)
.
.
.
.
7
7
7
8
8
4
General Contents:
4.1 API Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2 Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3 Release Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9
9
9
10
5
Indices and tables
19
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
i
ii
snakeoil Documentation, Release -trunk
Snakeoil is a library with the rough intentions of holding common python patterns/implementations, optimized common functionality, functionality to ease supporting multiple python versions (specifically python2.5 through 3.3), and
finally well tested implementations of rather hard problems to solve.
Snakeoil’s naming was chosen partially as a python analog of liboil (per cpu assembly optimizations of common operations), and also as a partial warning riffing on the common meaning of Snake Oil; specifically a supposed miracle cure
that has no effect. To be clear, this package does have some very powerful functionality and optimizations available,
and its usage is well known to speed things up: the pkgcore package manager from which this library was derived,
makes heavy usage of these optimizations- further in a simple test usage of snakeoil.caching.WeakInstMeta
to portage‘s Atom class, emerge -ep system was sped up by 7% with a 20% memory reduction- specifically via
just adding that metaclass to their Atom class, a one line change.
The point the authors are trying to make however is that it is not a magic solve all. Badly performant code will still be
badly performant code if it’s due to algorithmic idiocies- swapping in a snakeoil implementation for what affects 3%
of your runtime won’t do anything for the 97% that is due to a bad quadratic algorithm for example. Nor will using the
more crazy functionality we’ve got make your code necessarily any better to read- while snakeoil endeavours to make
its functionality simple, and to simplify consuming code, its usage does not magically make horrible code better.
That said, a python mantra is “we’re all adults here”. Snakeoil fully subscribes to this philosophy- the warnings
provided are merely to try and make it clear to people that snakeoil while powerful, if used in a thoughtless fashion is
not advisable.
Please note that snakeoil while relying on extensions in certain spots, does not require the extensions to function- it
will just fallback to native python implementations if it cannot find its extensions to use.
Contents
1
snakeoil Documentation, Release -trunk
2
Contents
CHAPTER 1
Community
Snakeoil now lives in full at https://github.com/pkgcore/snakeoil. Issue tracking, wiki, groups, source, are all available
from there.
For IRC, the core developers are generally accessible for bugs/questions via the freenode network in the #pkgcore
channel.
3
snakeoil Documentation, Release -trunk
4
Chapter 1. Community
CHAPTER 2
Getting the source (downloading releases or trunk)
Snakeoil vcs of choice is git, and our source can be checked out at https://github.com/pkgcore/snakeoil
All releases are available at https://github.com/pkgcore/snakeoil/releases., with release news available at releases.
As for dependencies, snakeoil basically just requires python2.6 and up.
5
snakeoil Documentation, Release -trunk
6
Chapter 2. Getting the source (downloading releases or trunk)
CHAPTER 3
Snakeoil intentions
Following is a rough breakdown of the core areas snakeoil aims to cover. This is not comprehensive- generally
speaking things that are useful have a way of winding up being added to snakeoil for reuse elsewhere.
3.1 Python annoyances and hard issues
The philosophy behind snakeoil is essentially that things should just work- to that end, and via a fair amount of unit
testing, snakeoil internals will cover up the nasty details of how to do something while presenting a simple/no surprises
api for consumers. Good examples of this are:
• snakeoil.weakrefs.WeakRefFinalizer, a metaclass allowing you to write __del__ methods without
the gc cycle issues inherent in the cpython implementation previous to 3.4.
• snakeoil.caching.WeakInstMeta, a metaclass allowing you to inline instance reuse for immutable objects. Essentially, why write multiple factory implementations? Why not push it directly into instance generation
itself so that the instance sharing is transparent, not requiring the consumer to know that it’s occuring?
• snakeoil.mappings.autoconvert_py3k_methods_metaclass; 2to3 translation will not rename
methods, leaving anyone implementing the mapping protocol in a tough situation- since py2k items differs
greatly from py3k items (py3k’s is essentially py2k’s iteritems), developers targeting both py2k/py3k are
in a tough spot. Specifically, they wind up having to write nasty if/else blocks into the code. The purpose of this
metaclass is to hide all of that nastiness into a single spot- you write your class targeting py2k, the metaclass
will rewrite the method layouts to match py3k at runtime if used in a py3k environment.
• snakeoil.obj.DelayedInstantiation(); a object proxy implementation that is effectively fully
transparent for any non-builtin target proxying it does. While proxying implementations exist, the authors
are aware of none that are reusable in this fashion, nor any that address the full issue of slotted methods (see the
module for full details).
For the issues described above, these are not the simplest problems to solve- people typically solve this on an adhoc
basis, partially solving the issue but never fully addressing it. The purpose of snakeoil’s functionality in that vein is to
solve it once and for all, in one spot, with the best possible implementation. Let developers worry about their problem
at hand, rather than worry about solving an issue someone else has already addressed essentially.
3.2 Supporting multiple python versions can be a pain
Another facet of snakeoil functionality is python compatibility- already mentioned was
snakeoil.mappings.autoconvert_py3k_methods_metaclass,
snakeoil.compatibility
is a separate module that exists to address compatibility issues across py2.7 to py3.4- whether it be intern moving
7
snakeoil Documentation, Release -trunk
to sys.intern, there is a significant amount of functionality in there to help with these issues. Note that compatibility
functionality that goes into that module isn’t the only compatibility bits- that’s just the general grab bag for it.
snakeoil.currying is another example (primarily targeting functools.partial, although providing more
functionality than just that limited usage).
3.3 Optimizations
Snakeoil provides a fair amount of optimized implementations. snakeoil.osutils is a good spot to start for
file operations/path manipulations, optimized sequence flattening (snakeoil.lists.iflatten_instance),
and optimized common patterns for class implementations (for example the snakeoil.klass.jit_attr()
decorator which converts secondary calls to the cached attribute into c level lookup speeds). There is a fair bit more
than just those examples- it’s in the readers interest to peruse our module docs, there is a fair amount available.
For importation speed issues, we provide snakeoil.demandload- for anyone who has used bzrlib.lazy_import
or hg’s equivalent implementation, this should be familiar. In effect, it allows you to delay importation till it’s actually
needed. While it’s not obvious, for well written scripts the time required for importation can be come a large problem
leaving people either the option of splitting up all functionality (which can be very problematic from a codebase
comprehension standpoint), or suffering the performance degradation.
As such, having a lazy importer is rather useful- your code still flows the same, just the import is done only when
something actually needs to access that functionality.
While we provide speed optimized implementations, we also provide functionality for reduction of memory usage- for
codebases with a large number of small dictionaries, snakeoil.obj.make_SlottedDict_kls() can reduce
the memory requirement in the range of 75-95%. For codebases that make extensive use of __slots__ for memory
reasons, it’s advised that they take a look at snakeoil.test.test_slot_shadowing
3.4 Avoiding Boilerplate (functionality to help with DRY- Don’t Repeat
Yourself)
Finally, for folks who have wrote a significantly large python codebase they wind up finding themselves repeatedly
writing the same type of functionality, over and over. Things like __eq__ methods, attribute/class attribute aliasing for backwards compatibility, JIT properties, and cloning documentation from preexisting sources. If interested,
snakeoil.klass is a good place to start reading.
8
Chapter 3. Snakeoil intentions
CHAPTER 4
General Contents:
4.1 API Documentation
4.2 Modules
snakeoil.bash
snakeoil.caching
snakeoil.caching_2to3
snakeoil.chksum
snakeoil.compatibility
snakeoil.containers
snakeoil.currying
snakeoil.data_source
snakeoil.demandload
snakeoil.dependant_methods
snakeoil.descriptors
snakeoil.distutils_extensions
snakeoil.fileutils
snakeoil.formatters
snakeoil.iterables
snakeoil.klass
snakeoil.lists
snakeoil.mappings
snakeoil.modules
snakeoil.obj
snakeoil.osutils
snakeoil.pickling
snakeoil.stringio
snakeoil.struct_compat
snakeoil.tar
snakeoil.test
snakeoil.test.mixins
snakeoil.unittest_extensions
snakeoil.version
snakeoil.weakrefs
snakeoil.xml
Functionality for reading bash like files
Transparent opportunistic weakref instance caching
python 2to3 with caching support
chksum verification/generation subsystem
Compatibility functionality for python 2.7 through 3.4
Container classes and functionality for implementing them
Function currying, generating a functor with a set of args/defaults pre bound.
A far more minimal form of file protocol encapsulation location and encoding into it
Demand load things when used.
Metaclass to inject dependencies into method calls.
Classes implementing the descriptor protocol.
collection of distutils extensions adding things like automatic 2to3 translation,
file related operations, mainly reading
Classes wrapping a file-like object to do fancy output on it.
Collection of functionality to make using iterators transparently easier
common class implementations, and optimizations
sequence related operations and classes
Miscellanious mapping related classes and functionality
dynamic import functionality
Object trickery including delayed instantiation and proxying
OS related functionality
convenience module using cPickle if available, else failing back to pickle.
StringIO functionality
Minor struct enhancements, and python version compatibility implementations
fixed up Tarfile implementation
Our unittest extensions.
collection of distutils extensions adding things like automatic 2to3 translation,
Version information (tied to git).
Optimized WeakValCache implementation, and a __del__ alternative
Compatibility code, preferring cElementTree, falling back as necessary.
9
snakeoil Documentation, Release -trunk
4.3 Release Notes
4.3.1 snakeoil trunk
• Remove pre-python2.7 compat support including iterables.chain_from_iterables (use chain.from_iterable from
itertools), is_disjoint (use isdisjoint), and built-in backports for all, any, and next.
• Drop deprecated currying.alias_class_method; use klass.alias_method.
• Migrate pkgcore.vdb.ondisk.bz2_data_source to snakeoil.data_source.bz2_source.
• Drop deprecated getters from snakeoil.data_source; attrs and functions should be accessed directly.
• Move snakeoil.fileutils.read_dict to snakeoil.bash.read_dict and drop compatibility shims for the iter_read_bash
and read_bash_dict methods from fileutils.
• Add support to klass.steal_docs to clone docstrings from regular functions in addition to class functions.
4.3.2 snakeoil 0.5.3 (2013-09-26)
• Simplify sphinx-build handling, removing checks for Gentoo specific suffixes.
• Switch from pbzip2 to lbzip2 for parallel bzip2 support since lbzip2 can handle parallel decompression of regular
non-lbzip2 compressed files unlike pbzip2.
• Fix python3.3 support.
4.3.3 snakeoil 0.5.2 (2012-10-17)
• Fixed doc generation for py3k, including threading appropriate python path/version down through the generation.
4.3.4 snakeoil 0.5.1 (2012-09-29)
• Fix corner case exception in formatter extension, cleanup potential NULL derefs.
• If hashlib has a whirlpool implementation, we use it and prefer it over mhash or our fallback implementation;
it’s faster, drops the GIL, and generally is the bees-knees.
• compatibility.raise_from no longer looses traceback information in >py3k.
4.3.5 snakeoil 0.5 (2012-08-04)
• lintplugins were updated to pylint 0.25.1 API; likely works with >=0.21.
• Added awareness of PEP3149 naming schemes to the namespace walkers.
• Fixed utime related race in 2to3 cacher; comes about due to python not stamping the inode w/ the exact float
given, represents via a particular source file being converted a second time (typically breaking it); only triggerable in local development, however it’s annoying, thus sorted.
• Effective immediately, python2.4 is no longer supported. Last release was in ‘08, so really, really don’t care
anymore.
• snakeoil.chksum grew whirlpool support, including native python fallback.
• snakeoil.chksum grew sha512 support.
10
Chapter 4. General Contents:
snakeoil Documentation, Release -trunk
• snakeoil.sphinx_utils was added w/ two reusable/importable scripts;
– generate_api_rsts.py: scans a given python namespace, generating properly structured ReST docs. This is
intended for better api doc generation than what sphinx’s autodoc tools currently provide.
– generate_news_rst.py: given a mostly ReST like NEWS/changelog file, this can convert into into pages
like http://docs.snakeoil.googlecode.com/git/news.html . Given appropriate args, it can bind in release
urls, git shortlog urls, and generally make it fairly pretty while useful.
• snakeoil.version is now reusable for other projects, and the _verinfo format it uses has been converted to storing
a dictionary (better deserialization namely via having it as a dict).
• snakeoil.distutils_extensions:
– sphinx_build_doc now always returns a class for usage, rather than None if sphinx wasn’t available. Clients
should use this, and structure their deps appropriately to not execute doc building unless desired (in which
case it’s better to have the command throw an error, instead of having distutils state “no such command”
for a build_doc target for example).
– build and install now support generating _verinfo files automatically from git, and installing them if the
support is enabled.
– All bzr related code has been ripped out in full.
• Docstring work, and general doc’s overhaul (including site updates).
• snakeoil.process now exposes functionality for finding the number of actual HW cores, filtering out HT cpus.
This is relevant since certain operations (pbzip2 in particular) aren’t any faster using HT- they just consume
more cpu.
• Api’s have been shifting a bit; compatibility was left in place, but large chunks of snakeoil.osutils and
snakeoil.fileutils have moved to the appropriate place.
• Compression framework was added; snakeoil.compression. Has built in awareness of pbzip2, parallelization,
and will use the most efficient form it can to get things done (primarily threaded, but implemented in a fashion
where the GIL doesn’t matter thus can easily hit multi core).
• closerange compatibility method was added for <2.6; this method of closing is far faster than normal “scan all
fds”, thus exposing it.
4.3.6 snakeoil 0.4.6 (2011-12-14)
• for bash parsing, pass into shlex the file being read so that relative source calls can find the file correctly. Issue
#1.
• add currying.wrap_exception and currying.wrap_exception_complex
4.3.7 snakeoil 0.4.5 (2011-11-30)
• Fix defaultdict in py2.4 to be compatible with >=py2.5 defaultdict.
• Fix WeakRefFinalizer so that instances that are still strongly referenced at the time of sys.exit have their finalizers ran via atexit; specifically, run the finalizers only for that pid.
4.3.8 snakeoil 0.4.4 (2011-10-26)
• use sane permissions for directories created for tests.
• swallow ENOTDIR from readfiles and readlines if told to ignore missing files.
4.3. Release Notes
11
snakeoil Documentation, Release -trunk
4.3.9 snakeoil 0.4.3 (2011-09-27)
• snakeoil.demandload is now threading aware, and no longer will complain if threading leads to an accidental
already-replaced placeholder access.
• snakeoil.osutils.read* moved to snakeoil.fileutils; compatibility shim will be removed in 0.5.
• fileutils.write_file was added for quick one off writes.
• chksums generation now will parallelize where worthwhile. Since this is python, GIL bouncing does occur,
ultimately costing more total CPU for the larger/more chksums. That said, it’s overall faster going in parallel
(for 4 chksummers, it’s about 75% faster; for 2, about 40% faster).
Again, note this is enabled by default. To disable, parallelize=False.
• added snakeoil.process for getting processor count
• don’t install compatibility_py3k.py if we’re targetting py2k; no need, and it pisses off pyc generation.
4.3.10 snakeoil 0.4.2 (2011-09-02)
• compatibility.raise_from; compatibility across py2k/py3k for doing py3k raise EXCEPTION from CAUSE; see
pep3134. Primarily for raising an exception which was caused by another (casting an exception essentially).
• added klass.cached_property, and fixed klass.jit_attr to block bad usage that goes recursive.
• add distutils_extension for building sphinx docs
• if the invoking python has issue 7604 fixed, then use a fast single lookup version of delitem for slotted instances;
else use the normal double lookup workaround.
4.3.11 snakeoil 0.4.1 (2011-06-22)
• issue 7567; python2.7.1 reintroduces it (2.7 lacked it). Gentoo bug 350215.
• snakeoil.unittest_extensions was split out from distutils_extensions.
• snakeoil.obj.make_SlottedDict_kls moved to mappings; it’ll be removed from snakeoil.obj in 0.5.
• currying.alias_class_method is now deprecated; use klass.alias_method instead.
• handle differing lib2to3 dependant on multiprocessing existance.
4.3.12 snakeoil 0.4 (2011-04-24)
• added snakeoil.klass.immutable_instance metaclass and an equivalent inject function for modifying the
scope. These are used to avoid classes adhoc’ing the same sort of functionality, rarely throwing appropriate/standardized exceptions.
• for any consumers of snakeoil’s common header, for py2.4/py2.5 we’ve added suppression of the segfault potential for Py_CLEAR(tmp); see http://mail.python.org/pipermail/python-bugs-list/2008-July/055285.html for the
sordid details.
• mappings.inject_getitem_as_getattr, and AttrAccessible were added. The former is for modifying a class so that
attribute access is proxied to item access (including rewriting KeyError to AttributeError); the latter is a general
usable class for this.
• mappings.ListBackedDict and mappings.TupleBackedDict have been removed.
12
Chapter 4. General Contents:
snakeoil Documentation, Release -trunk
• demandload.demand_compile_regexp no longer returns the placeholder- instead it injects the placeholder directly into the scope, just like demandload does.
• added snakeoil.iterables.chain_from_iterable; this is compatibility for py2.4/py2.5, in >=py2.6 it just uses itertools.chain.from.iterable .
• initial work towards jython2.5 support.
• Massive amount of docstring work. Yes, snakeoil is now documented and has examples.
• correct an off by one in caching_iter.
• snakeoil.dependant_methods.ForcedDepends grew two new methods; __set_stage_state__ for tweaking stage
state manually, and __stage_step_callback__ for being notified on each stage completed.
• snakeoil.stringio; basically a py2k/py3k compatible set of class wrapping cStringIO/StringIO as necessary to
provide readonly or writable versions of text vs bytes StringIO handles. Note that readonly instances throw
TypeError on write/truncate/etc, instead of cStringIO’s behaviour or just not having the methods (or silently
modifying things).
• pkgcore ticket 172; posix access technically allows for a root invoker to get a True result when doing X_OK
on a non-executable file; this renders the function a fair bit useless for doing $PATH lookups for example, so
we bundle a native python implementation that is fallen back to for userlands (opensolaris for example) that
choose to implement that broken posix option. Linux/*BSDs don’t have this issue, so os.access is used for those
userlands.
• pkgcore ticket 13; data_source.get* functions return handles that have .exceptions holding the exceptions they
can throw, and that are caused by underlying implementation issues (versus caused by bad usage of the object).
• snakeoil data_source’s will loose their get_ methods in the next major version- they’re kept strictly for compatibility.
• fix_copy.inject_copy will be removed after the next major version. What remains does nothing.
• pkgcore.chksum was moved to
snakeoil.data_source in addition.
snakeoil.chksum;
pkgcore.interfaces.data_source
was
moved
to
• all bash functionality was split out of .fileutils into .bash
• osutils.readlines arg strip_newlines became strip_whitespace; if set, it’ll wipe all leading/trailing whitespace
from a line.
• snakeoil.weakrefs grew a new experimental metaclass; WeakRefFinalizer. Basically this class allows __del__
without the GC issues __del__ normally suffers. Experimental, but should work- just keep in mind you get
proxies back from users of that class.
• snakeoil.test.test_del_usage was added to scan for classes using __del__ when they could use WeakRefFinalizer
instead.
• snakeoil.lists.predicate_split; given a predicate function, a stream, and an optional key function (think DSU
pattern for sorted), split the stream into two sequences- one sequence where the predicate evalutes true, the
other sequence where it evaluates false.
• detect python bug 3770 (gentoo bug 330511), and disable multiprocessing for 2to3 conversion if it’s found.
4.3.13 snakeoil 0.3.7 (2010-06-26)
• detect python bug 4660, and disable parallelization in 2to3 conversion if the system suffers from it. This fixes
an occasional “task_not_done” ValueError.
• minor optimization to TerminfoFormatters to cache and reuse TerminfoColor. Exempting the formatter, Terminfo* objects are now immutable
4.3. Release Notes
13
snakeoil Documentation, Release -trunk
• snakeoil.mappings.defaultdict; compatibility implementation, defaults to collections.defaultdict for >=python2.5, a native python implementation for 2.4
4.3.14 snakeoil 0.3.6.5 (2010-05-21)
• add discard method to AtomicWriteFile to intentionally discard the updated content.
• fix initialization of RefCountingSet to set the refcount correctly on duplicate keys
4.3.15 snakeoil 0.3.6.4 (2010-04-21)
• fix rare segfault potential with cpython generic_equality __eq__/__ne__ when it’s blindly transferred across
classes.
• fix py3k handling of terminfo entries- xterm for example was affected.
4.3.16 snakeoil 0.3.6.3 (2010-03-14)
• ‘dumb’ terminfo is no longer tempted- to useless to hack around it.
• get_formatters now properly falls back to plain text formatting if no terminfo could be found.
4.3.17 snakeoil 0.3.6.2 (2010-02-15)
• overhauls to 2to3k support; speedup caching by near 16% via moving it into the process rather then as an
external invocation. Additionally fork the workers off to # of cpus on the system for parallelization when the
results aren’t cached.
• force -fno-strict-aliasing to be appended when it’s invalidly left out by distutils internals. See issue 969718 in
pythons tracker. If you’re using a non gcc compiler, you’ll need to pass –disable-distutils-flag-fixing to disable
the -fno-strict-aliasing additions.
4.3.18 snakeoil 0.3.6.1 (2010-02-07)
• Licensing changes- see COPYING for specifics. Majority of snakeoil is now GPL2/BSD 3 clause w/ a few
exemptions.
• minor cleanup to extensions for GC support and stricter gcc.
4.3.19 snakeoil 0.3.6 (2010-01-08)
• add a cpy extension for jit_attr functionality; this brings the overhead down to effectively background noise for
most usages.
• add a reflective_hash class to snakeoil.klass; this is primarily used for when the has is precomputed and stored
somewhere.
• add an extension for ProtectedSet.__contains__; this levels a nice speedup for pcheck scans.
• enable a set of extensions for slots backed mappings; primarily affects pkgcore cache data objects, end result
being pquery against a full repo in raw mode is about 8% faster overall.
14
Chapter 4. General Contents:
snakeoil Documentation, Release -trunk
4.3.20 snakeoil 0.3.5 (2009-12-27)
• snakeoil.struct_compat module was added; provides py2.4 compat, and adds read/write methods that take an fd
and operate as unpack/pack against that fd. This simplifies invocation/stream access primarily.
• add test_slot_shadowing; basically looks for __slots__ usage where a derivative class adds slotting the parent
already provides, thus leading to a very unfun set of bugs and wasted memory.
• fix test_demandload_usage to properly recurse...
4.3.21 snakeoil 0.3.4 (2009-12-13)
• add compatibility.is_py3k_like for marking if it’s >=py2.7, or py3k
4.3.22 snakeoil 0.3.3 (2009-10-26)
• use the registration framework for epydoc to make it aware of partials.
• monkeypatch pydoc.isdata on the fly to be aware of partials. This makes pydoc output far more useful (and
matches what is expected).
• experimental py3.1 support via 2to3. setup.py automatically will convert the source if invoked by a py3k interpretter.
• snakeoil.osutils.readlines was expanded out into multiple functions, utf8, ascii, utf8_strict, ascii_strict, and
bytes. ‘Strict’ means that we always want it decoded. Non strict is useful when the file has some utf8 in it
you don’t care about, and don’t want to take the codecs.open performance hit under py2k. Under py3k, it’s
always decoded (required due to py3k changes).
• snakeoil.osutils.readfile was expanded out into multiple functions, utf8, ascii, ascii_strict, and bytes. Use the
appropriate one- this will make py3k compliance far easier.
• optimization in snakeoil.osutils.readlines; for small files, it’s roughly a 4-8% speedup, for larger files (over half
a meg) growing past 25%. This puts it’s performance at roughly 2x over the open equivalent for small files, and
near 10-15% faster for larger files.
• snakeoil.klass grew new properties to ease common tasks; jit_attr (invoke the target func to get the value, cache
the value, return that value till the cached value is wiped). alias_attr (when that attr is accessed, hand the attribute
the alias targets).
• snakeoil.compatibility additions; next, cmp, file_cls, and is_py3k, next, intern, sort_cmp (to paper over sorted
no longer accepting a cmp arg), and sort_cmp (to paper over list.sort no longer accepting a cmp arg).
• snakeoil.klass.cached_hash; decorator to automatically cache the results of the target function. primarily intended for __hash__ implementations.
• snakeoil.klass.inject_richcmp_methods_from_cmp ; passed a class scope, it’ll automatically add __le__, __lt__,
__gt__, __eq__, etc, via invoking __cmp__ if the python version is py3k.
• snakeoil/caching_2to3.py, a caching form of 2to3 that relies on an env var ‘PY2TO3_CACHEDIR’ to determine
where to store cached versions of converted source. Algorithm behind the cache is md5 based- if the md5 of
the targeted source exists in the cachedir, it reuses the results from the previous run instead of invoking 2to3.
Massive performance speed up from this- uncached, setup.py test is ~32s. cached, ~1.9s. That said, this is
experimental- bug reports welcome however.
• setup.py test has been heavily enhanced- now it does it’s testing against a standalone install of the source, should
have zero side affects on the underlying source.
4.3. Release Notes
15
snakeoil Documentation, Release -trunk
• paper over a bug in cElementTree where it fails to import fully, but doesn’t raise ImportError. This address
upstream python bug 3475.
• snakeoil no longer installs a bundled copy of elementtree if the python version is 2.5 or higher (no need, python
bundles it’s own).
• snakeoil.test.test_demandload_usage now supports blacklisting- this is primarily useful for blocking py3k specific modules from being checked under py2k, and vice versa.
• in test_demandload_usage helper functionality it’s possible for a file to disappear under it’s feet- ignore it, lock
files from trial can trigger this. Note it via logging.warn, and continue.
4.3.23 snakeoil 0.3.2 (2009-03-24)
• handle a race condition in ensure_dirs where the directory is created underfoot (thus a non issue).
• massive memory reduction for snakeoil.tar monkey patching; via punting the unused .buf storage (512 byes per
TarInfo). Grand total, this is a 70% reduction of the memory used compared to vanilla TarInfo (50% less then
snakeoil 0.3).
• tweak snakeoil.tar monkey patching to re-enable memory savings on python2.6
• correct python2.6 compatibility issues; __(sizeof|format|subclasshook)__ awareness, and handle getattr throwing AttributeError in the infinite recursion getattr tests.
• for test_demandload_usage, output the exception that caused the demandload ‘touch’ to fail.
4.3.24 snakeoil 0.3.1 (2008-11-07)
• pkgcore ticket 215; fixup corner case errors in normpath cpy.
4.3.25 snakeoil 0.3 (2008-08-28)
• refactor dependant_methods to stop creating strong cycles that the python vm seems unable to break. Shift the
func storage away from .raw_func to .sd_raw_func in addition. Add in __(un|)wrap_stage_dependencies__ so
that invocation of unwrap then wrap will make changes to stage_depends take affect.
• intern gname and uname for TarInfo objects via property trickery- again, purpose being less memory usage.
• AtomicFile now marks itself as initially finalized until it has a fd; this removes spurios complaints from __del__
• LimitedChangeSet got an additional kwarg; key_validator. A function can be passed in via this to do validation
of the desired key- either it throws an exception, or returns the key to use.
4.3.26 snakeoil 0.2 (2008-03-18)
• snakeoil.fileutils.iter_read_bash and friends grew an allow_inline_comment param to control stripping of inlined
comments; defaults to True.
• bash parsing bug where “x=y” w/out a trailing newline wasn’t returning the ‘y’ value.
• x=-* (specifically unquoted) is a valid assignment, fixed.
• added SNAKEOIL_DEMANDLOAD_PROTECTION environment variable- if set to something other then
‘yes’, disables the placeholder checks. Main intention for this functionality is for when code is introspecting
demandload consuming code (epydoc for example), and inadvertantly triggers the access multiple times.
16
Chapter 4. General Contents:
snakeoil Documentation, Release -trunk
4.3.27 snakeoil 0.1 (2007-11-11)
• Add a cpython version of snakeoil.formatters.
4.3.28 snakeoil 0.1-rc2 (2007-07-06)
• Pulled in any/all cpy extensions if not available in current python version.
• Added several pylint checks for naughty things like bool(len(seq)), itering over dict.keys() and shadowing
builtins.
• Misc doc improvements.
• Rewrite demandload with a new multiple arg style, and update the appropriate pylint checker.
• Fix title updating by flushing the formatter’s stream.
• overhaul demandload test case for consuming code.
• Add snakeoil.containers.SetMixin to provide set methods for various objects.
• Remove snakeoil.const - unused.
• Improve test coverage in general.
• Add folding dicts.
• Move snakeoil.file to snakeoil.fileutils.
• Initial release, split out from pkgcore.util.*.
4.3. Release Notes
17
snakeoil Documentation, Release -trunk
18
Chapter 4. General Contents:
CHAPTER 5
Indices and tables
• genindex
• modindex
• search
19