Brad's Blog

Something I should probably have noticed before but Amazon S3 list_bucket responses can be truncated if theres a lot of entries in a bucket.

If truncation has occurred the is_truncated attribute is set to be true on the response.

Follow-on requests can be made by setting the marker option to the call to list_bucket to be the key of the last entry received:

1  options = { 'marker' : list_bucket_response.entries[-1].key }
2  list_bucket_response = conn.list_bucket(bucket, options)

Here's the release of AmazonS3Store 1.1 with the changes: AmazonS3Store-1.1.tar.gz.

Version 1.1 - Fri Mar 28 00:02:04 GMT 2008
 -----------------------------------------
 * Added internal function __list_bucket_iter(cmdopts, cfg, conn)
   which wraps the connection list_bucket. The function is an iterator and
   makes multiple requests where truncated responses are detected.

This one caught somewhat by surprise - mostly because of the unusual use of a 500 (Internal Server Error) code by LWP::UserAgent to flag a client-side configuration problem.

The answer, as with so many things perl, lay with the PerlMonks in this article: Mysterious LWP status 500 Line too long.

To summarise: Net::HTTP has a default maximum header line length of 4096. This can be changed by passing a MaxLineLength option to the constructor - and happily LWP::UserAgent has an internal hash which can be tweaked to set the desired maximum line length:

1  my %OPTS = @LWP::Protocol::http::EXTRA_SOCK_OPTS;
2  $OPTS{MaxLineLength} = 8192; # Or however large is needed...
3  @LWP::Protocol::http::EXTRA_SOCK_OPTS = %OPTS;

__init__ changed between versions 2.4 and 2.5 of python - in 2.5 keyword arguments are available.

I've just used the old-style call which is supported in 2.5:

1  -    imported_module = __import__(module_fqname, fromlist = [1])
2  +    imported_module = __import__(module_fqname, globals(), locals(), [1])

Version 1.3 - Sun Mar 16 11:24:14 GMT 2008
 -----------------------------------------
 * Changed calls to __import__ to be compatible between python versions
   2.4 and 2.5 (in 2.5 __import__ supports keyword arguments)

pyprove-1.3.tar.gz

And now another release for a bugfix:

Version 1.2 - Fri Mar 14 21:57:01 GMT 2008
 -----------------------------------------
 * When doctest.DocTestSuite throws a ValueError (has no tests)
   recover from the exception and move on

pyprove-1.2.tar.gz

A full several minutes between the first and second release, not too bad...

As per the changelog:

Version 1.1 - Fri Mar 14 19:36:15 GMT 2008
 -----------------------------------------
 * Change appending of library_prefix_paths entries to
   inserting them at the beginning of sys.path to make sure
   local source is found before source installed other places
   in the path

 * Pre-pend os.getcwd() to sys.path to help finding libraries
   in the right place without having to set PYTHONPATH

Here it is: pyprove-1.1.tar.gz

I know this exists in various guises around the place but I've been coming back to wanting something easy to use like perl prove when writing python.

A while back I was doing some work on Zope and they have test.py which does just that - but it's somewhat specific to Zope.

So I've sat down and come up with pyprove - the idea being that it should be possible to run the script within a source code tree and have it go out and find unittest and doctest tests and run them all.

Thus far it's pretty simple - no command line options are available and it only runs from the current directory. It also requires that PYTHONPATH is set if tested modules are not in the current search path.

Here's a source distribution: pyprove-1.0.tar.gz.

(Debugging is greatly facilitated by re-including the commented out logging configuration line.)