Sat Jun 21 20:01:02 BST 2008

Building up to a fully defined FUSE class

I've been playing around with FUSE on and off since my inital post and the real killer for me has been finding out what is failing when it fails (for the same reasons that I ended up wrapping all of my methods in exception catchers to log problems somewhere I could see them).

Developing a filesystem incrementally (as I have been) the problem is that sometimes FUSE starts to make calls to methods that I have not yet defined. As these calls usually just result in a silent failure all I end up seeing is a failed operation without any clue as to why that might be.

The approach I've now settled upon is to register all of the operations named in the FUSE API with a reporting stub:

 1  import fuse
 2  import logging
 3  import inspect
 4  import pprint
 5
 6  def _undef_sub(name):
 7    def handler(*args, **kwargs):
 8      stack_string = pprint.pformat(inspect.stack())
 9      logging.debug('Undefined method[%s] called on stack: %s' % (name, stack_string))
10    return handler
11
12  class MyFs(fuse.Fuse):
13    def __init__(self, *args, **kw):
14      fuse.Fuse.__init__(self, *args, **kw)
15      logging.basicConfig(level     = logging.DEBUG,
16                          format    = '%(asctime)s %(levelname)-8s %(message)s',
17                          datefmt   = '%a, %d %b %Y %H:%M:%S',
18                          filename  = '/tmp/debug.log',
19                          filemode  = 'a')
20      logging.debug('MyFs object instanciated')
21
22      ### Not yet defined methods
23      self.getattr      = _undef_sub('getattr')
24      self.readlink     = _undef_sub('readlink')
25      self.mknod        = _undef_sub('mknod')
26      self.mkdir        = _undef_sub('mkdir')
27      self.unlink       = _undef_sub('unlink')
28      self.rmdir        = _undef_sub('rmdir')
29      self.symlink      = _undef_sub('symlink')
30      self.rename       = _undef_sub('rename')
31      self.link         = _undef_sub('link')
32      self.chmod        = _undef_sub('chmod')
33      self.chown        = _undef_sub('chown')
34      self.truncate     = _undef_sub('truncate')
35      self.utime        = _undef_sub('utime')
36      self.open         = _undef_sub('open')
37      self.read         = _undef_sub('read')
38      self.write        = _undef_sub('write')
39      self.statfs       = _undef_sub('statfs')
40      self.flush        = _undef_sub('flush')
41      self.release      = _undef_sub('release')
42      self.fsync        = _undef_sub('fsync')
43      self.setxattr     = _undef_sub('setxattr')
44      self.getxattr     = _undef_sub('getxattr')
45      self.listxattr    = _undef_sub('listxattr')
46      self.removexattr  = _undef_sub('removexattr')
47      self.opendir      = _undef_sub('opendir')
48      self.readdir      = _undef_sub('readdir')
49      self.releasedir   = _undef_sub('releasedir')
50      self.fsyncdir     = _undef_sub('fsyncdir')
51      self.init         = _undef_sub('init')
52      self.destroy      = _undef_sub('destroy')
53      self.access       = _undef_sub('access')
54      self.create       = _undef_sub('create')
55      self.ftruncate    = _undef_sub('ftruncate')
56      self.fgetattr     = _undef_sub('fgetattr')
57      self.lock         = _undef_sub('lock')
58      self.utimens      = _undef_sub('utimens')
59      self.bmap         = _undef_sub('bmap')

I was originally going to try and extract the method name automagically but it proved to be much simpler to just assign the name explicitly. As methods are implemented properly the _undef_sub version can be dropped, leading slowly to a full implementation with much fewer headaches!.

The use of inspect and pprint came from here and here.

Comment by: Bradley Dean at Tue, 24 Jun 2008 23:05:31 (BST)

Better yet - this version of _undef_sub provides more information about what is going on:

 1  def _undef_sub(name):
 2    def handler(*args, **kwargs):
 3      import inspect
 4      import pprint
 5      stack_string = pprint.pformat(inspect.stack())
 6      logging.critical('Undefined method[%s]' % name)
 7      if len(args) > 0:
 8        logging.critical(' .. Positional args: %s' % `args`)
 9      if len(kwargs) > 0:
10        logging.critical(' .. Named args: %s' % `kwargs`)
11      logging.critical(' .. Called on stack: %s' % stack_string)
12    return handler

Posted by Bradley Dean | Permalink

Wed Jun 18 18:25:03 BST 2008

Bulk bookmark changes in del.icio.us using the API

I thought I'd try out del.icio.us as a possible roaming bookmark option today - and having tried out the import process I discovered that there's no way (on the web interface) to make bulk changes to your bookmarks.

Having decided to delete everything I had just imported I started playing around with the API - an excercise in patience given the somewhat overzealous throttling of requests. As an aside the https://api.del.icio.us/v1/posts/all request for all posts is much more sensitive to throttling - and using the method more than a few times in a short time seems to result in a block for 20-30 minutes.

I noticed that the advice for deleting all bookmarks is to delete and recreate your account - as per: http://del.icio.us/help/faq#How_do_I_delete_all_my_bookmarks - this was the exact answer to what I needed but the question of bulk-changes was still of interest so I persisted with the API.

The following script steps through all of the bookmarks and tries to delete them:

 1  #!/usr/bin/env python
 2
 3  import time
 4  import urllib
 5  import urllib2
 6  import xml.dom.minidom
 7  import getpass
 8
 9  # API urls used by this script
10  BASE_URL = 'https://api.del.icio.us/v1'
11  ALL_URL  = '%s/posts/all' % BASE_URL
12  GET_URL  = '%s/posts/get' % BASE_URL
13  DEL_URL  = '%s/posts/delete?url=' % BASE_URL
14
15  # As per http://del.icio.us/help/api/, del.icio.us indicate that
16  # requests should be delayed at least one second (in practice I
17  # found that a one second delay led to beign throttled, but two
18  # seconds was ok)
19  REQUEST_DELAY = 2
20
21  # This delay is used when the server responds with a 503 error (to
22  # indicate that you have been throttled)
23  ERROR_DELAY = 60
24
25  def retry_urlopen(url):
26    """ Handle throttling by waiting and retrying """
27    while True:
28      try:
29        response = urllib2.urlopen(url)
30        return response
31      except Exception, e:
32        print "Request failed to [%s] because [%s] ..." % (url, e),
33        print "Sleeping [%d] and retrying" % ERROR_DELAY
34        time.sleep(ERROR_DELAY)
35
36  def get_credentials():
37    """ Get the username and password and configure urllib2 """
38    username = raw_input('Enter del.iciou.us username: ').strip()
39    password = getpass.getpass('Enter del.iciou.us password: ').strip()
40
41    auth_handler = urllib2.HTTPBasicAuthHandler()
42    auth_handler.add_password( realm  = 'del.icio.us API'
43                             , uri    = BASE_URL
44                             , user   = username
45                             , passwd = password
46                             )
47    opener = urllib2.build_opener(auth_handler)
48    urllib2.install_opener(opener)
49
50  def get_all_bookmarks():
51    """ Retrieve all bookmarks for user """
52    time.sleep(REQUEST_DELAY)
53    response = retry_urlopen(ALL_URL)
54    return xml.dom.minidom.parseString(response.read())
55
56  def get_next_bookmarks():
57    """ Retrieve next bookmarks for user """
58    time.sleep(REQUEST_DELAY)
59    response = retry_urlopen(GET_URL)
60    return xml.dom.minidom.parseString(response.read())
61
62  def delete_bookmarks(bookmarks_dom):
63    """ Delete bookmarks found in response dom """
64    for post_element in bookmarks_dom.getElementsByTagName('post'):
65      time.sleep(REQUEST_DELAY)
66      post_url = post_element.getAttribute('href')
67      post_description = post_element.getAttribute('description')
68
69      print_line = "Deleting %s [%s]" % (post_description, post_url)
70      print print_line.encode('ascii','ignore')
71
72      retry_urlopen("%s%s" % (DEL_URL, post_url))
73
74  if __name__ == '__main__':
75    get_credentials()
76
77    all_bookmarks_dom = get_all_bookmarks()
78    delete_bookmarks(all_bookmarks_dom)
79
80    # If you've been throttled on requests to get all postcodes
81    # you can sometimes still access the bookmarks one at a time (leading
82    # to more requests to the server - but hey, it's their rules)
83    #bookmarks_dom = get_next_bookmarks()
84    #while len(bookmarks_dom.getElementsByTagName('post')) > 0:
85    #  delete_bookmarks(bookmarks_dom)
86    #  bookmarks_dom = get_next_bookmarks()

I used this script successfully (well - more or less, some urls did not match so a few manual deletes were necessary) before deciding that del.icio.us was not for me and deleting my account. :)

I won't be using del.icio.us - the layout doesn't really work for me so well as just copying my bookmarks.html file around - so I'll be sticking with that until I find what I'm after. In the mean-time it was an interesting excercise in using the API.


Posted by Bradley Dean | Permalink

Mon Jun 16 16:55:39 BST 2008

Displaying process IDs in Windows perfmon.exe counter setup

I recently found myself using windows perfmon.exe. I've been setting up trackers to monitor resource usage. The problem I've had is that the entries in the Select instances from list: list show processes by name, and where multiple processes have the same name they are given numeric suffixes, ie:

  • perl
  • perl#1
  • perl#2

It turns out there is a way to actually display the process id in this list - described here:

The Process object in Performance Monitor can display Process IDs (PIDs)

To summarise - set up a new DWORD registry value at

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PerfProc\Performance\ProcessNameFormat

and give it a value of 2 (enables PID) or 1 (disables PID).


Posted by Bradley Dean | Permalink

Sun Jun 1 22:49:45 BST 2008

Perl AUTOLOAD primer

Whenever I need to use the Perl AUTOLOAD I seem to spend the same few moments reminding myself of the basics (and then going back and fixing up something I had forgotten). This is a quick primer (in the form of a block of code) which should short-circuit that process.

Perl autoloading is described in the perldoc here:

This AUTOLOAD is for a class package - though the basic ideas are still valid for non-class packages.

 1  package Person;
 2
 3  use strict;
 4  use warnings;
 5
 6  # NEXT::ACTUAL will be used instead of SUPER so that the full
 7  # inheritance tree is searched for other AUTOLOADS. See
 8  # NEXT perldoc for the details.
 9  # NEXT::ACTUAL did not work for explicit calls to AUTOLOAD
10  # until version 0.61 (almost released on CPAN at the time
11  # of this article)
12  use NEXT 0.61;
13
14  # These are the valid class attributes
15  my @attrs = qw{ name age height address };
16
17  sub new {
18    my ($class, @params) = @_;
19    my $self = bless {}, $class;
20    $self->init(@params);
21    return $self;
22  }
23
24  sub init {
25    my ($self, @params) = @_;
26    $self->NEXT::init(@params);
27    $self->{$_} = undef for @attrs;
28  }
29
30  # If can is not overwritten the AUTOLOAD methods 
31  # will return false which might be misleading.
32  sub can {
33    my ($self, $method_name) = @_;
34
35    if ( grep { $_ eq $method_name } @attrs ) {
36      return 1;
37    }
38    else {
39      return $self->NEXT::can($method_name);
40    }
41  }
42
43  # This is a attribute accessor used by AUTOLOAD
44  sub _accessor {
45    my ($self, $attr, $value) = @_;
46
47    if ( exists $self->{$attr} ) {
48      if ( defined $value ) {
49        # Set value
50        $self->{$attr} = $value;
51        return $value;
52      }
53      else {
54        # Get value
55        return $self->{$attr};
56      }
57    }
58    else {
59      die "Invalid attribute name: ${attr}";
60    }
61  }
62
63  # When using AUTOLOAD, DESTROY needs to be handled by
64  # either the AUTOLOAD function or explicitly as is done
65  # here.
66  sub DESTROY { }; # no-op
67
68  # $AUTOLOAD is automatically set to be the fully-qualified
69  # name of the function being called
70  our $AUTOLOAD;
71
72  sub AUTOLOAD {
73    my ($self, @params) = @_;
74
75    # Commonly defined to alert incorrect usage
76    die 'not a object ref' unless ref $self;
77
78    # The method name is the final part of the string after the
79    # last : character
80    my ($method_name) = $AUTOLOAD =~ /([^:]*)$/;
81
82    # Accessor
83    if ( grep { $_ eq $method_name } @attrs ) {
84      @_ = ($self, $method_name, @params);
85      goto &_accessor;
86    }
87    # Unknown - pass to other classes
88    else {
89      # Fall through to super-classes if defined
90      # NEXT::ACTUAL should cause an exception if no AUTOLOAD
91      # is found elsewhere in the inheritance tree
92      return $self->NEXT::ACTUAL::AUTOLOAD(@params);
93    }
94  }

Note - at the time of writing I'm having difficulties getting NEXT::ACTUAL to fail as desired. If no AUTOLOAD is found the error seems to be failing silently. I'm discussing this with my fellow Melbourne Perl-Mongers so should hopefully have a resolution shortly.

Comment by: Bradley Dean at Mon, 2 Jun 2008 02:08:16 (BST)

Looks like I spotted a bug in NEXT - having discussed this with Damian it looks like we may have a fix in the works:

Comment by: Bradley Dean at Wed, 4 Jun 2008 10:35:54 (BST)

Version 0.61 (almost on CPAN as of this date) includes the fix for calling NEXT::ACTUAL::AUTOLOAD.


Posted by Bradley Dean | Permalink