Brad's Blog

Having written Finance::Quote::currency_lookup I was able to get back to the little script that triggered the idea in the first place

The idea: a script which I can call with partial code or currency names and an amount which will then try and tell me what the current converted value is. If there's something other than a single matching currency let me know that nothing or too many things matched.

Here's the script (also attached):

 1  #!/usr/bin/env perl
 2
 3  use strict;
 4  use warnings;
 5
 6  use Finance::Quote;
 7  my $q = Finance::Quote->new();
 8
 9  my %trans = ();
10  @trans{qw(from to)} = @ARGV;
11
12  # Copied this patter from Finance::Quote::currency to be consistent
13  $trans{from} =~ s/^\s*(\d*\.?\d*)\s*//;
14  my $amount = $1 || 1;
15
16  # Search out currencies
17  for my $key (qw( from to )) {
18    # Is it a code or a name?
19    my $currencies;
20    if ( $trans{$key} =~ /^[A-Z]{2,}$/ ) {
21      # Guess code
22      $currencies = $q->currency_lookup( code => qr/$trans{$key}/ );
23    }
24    else {
25      # Guess name
26      $currencies = $q->currency_lookup( name => qr/$trans{$key}/i );
27    }
28    if ( scalar keys %{$currencies} == 1 ) {
29      my $real_key = "real_${key}";
30      ($trans{$real_key}) = keys %{$currencies};
31      printf "%s : %s (%s)\n", $key
32                             , $trans{$real_key}
33                             , $currencies->{$trans{$real_key}}->{name};
34    }
35    elsif ( scalar keys %{$currencies} > 1 ) {
36      print "Multiple currency matches for ${key}:\n";
37      print " * $_ (" . $currencies->{$_}->{name} . ")\n" for keys %{$currencies};
38    }
39    else {
40      print "No currency matches for ${key}: " . $trans{$key} . "!\n";
41    }
42  }
43
44  # If real from and to exist run the conversion
45  if ( exists $trans{real_from} && exists $trans{real_to} ) {
46    print "${amount} $trans{real_from} => "
47        . $q->currency("${amount}$trans{real_from}", $trans{real_to})
48        . " $trans{real_to}\n";
49  }

And here's some examples of this script in action:

# Some Australian dollars to pounds
# (name searches only)
$ ./conv.pl 42aus brit
from : AUD (Australian Dollar)
to : GBP (British Pound)
42 AUD => 18.8454 GBP

# And the some pounds to US dollars
# (a name and a code search, because US is all-caps)
$ ./conv.pl brit US
from : GBP (British Pound)
to : USD (U.S. Dollar)
1 GBP => 1.4225 USD

# Oops - a typo on pounds
$ ./conv.pl GPB euro
No currency matches for from: GPB!
to : EUR (Euro)

# And now, to make the British cry...
$ ./conv.pl GBP euro
from : GBP (British Pound)
to : EUR (Euro)
1 GBP => 1.1317 EUR

# And finally, too many matches:
$ ./conv.pl egypt ven
from : EGP (Egyptian Pound)
Multiple currency matches for to:
 * SIT (Slovenian Tolar)
 * VEB (Venezuelan Bolivar)

While using Finance::Quote::currency() to do some conversions I found myself having to go searching around the web for currency codes. At the time I thought it would be much easier if you could just asked the module what currency codes existed - and so I started work on a currency_lookup() function to do just that.

Finance::Quote uses the Yahoo Currency Converter website so the new function needed to report a compatible list of currencies.

Having developed the search method against a statically stored list of currencies (as I hadn't implemented the live extraction yet) I then discussed options with the other developers and decided that a hard coded list was a better approach given that the list wasn't going to change very often.

Here's a basic idea of usage:

1    $currencies_by_name = $q->currency_lookup( name => 'Australian' );
2    $currencies_by_code = $q->currency_lookup( code => qr/^b/i      );
3    $currencies_by_both = $q->currency_lookup( name => qr/pound/i
4                                             , code => 'GB'         );

The return value is a hash of hash-refs - for example for the first query above:

1    $VAR1 = { AUD => { name => "Australian Dollar" } }

I had added a module to contain the currency list (Finance::Quote::Currencies). Once the search code was complete I then added a test to compare the static list with a live list and implemented Finance::Quote::Currencies::fetch_live_currencies(). This function uses HTML::Parser to extract the currency list. If the currency list changes this function can be used to easily updated the stored list.

These changes are currently available on the currency_lookup branch on github. Coming to a CPAN release in your town real soon now.

Question: How to use my wiki to store and test bits and pieces of plain html?

Answer: It should be simple, but there's no such formatter in MoinMoin (and no way I know to just say 'embed this text as-is') but surely this must be one of the simplest MoinMoin Parsers that could be written - so let's have a look...

The CSV Parser looked quite simple - from there it became clear that all you really need to do is initialise a couple of attributes in the constructor and then write the simplest-case format method:

 1  class Parser:
 2      """ Format HTML as HTML
 3      """
 4
 5      def __init__(self, raw, request, **kw):
 6          """ Store the source text.
 7          """
 8          self.raw = raw
 9          self.request = request
10          self._ = request.getText
11
12      def format(self, formatter):
13          """ Send it back as it came
14          """
15          self.request.write(self.raw)

Load this into your wiki plugin directory as data/plugin/parser/html.py directory, start a document with #FORMAT html and it just works. This probably wouldn't be a great feature for a public wiki, but it's nice to be able to roll your own so easily. :)

I ran into a cross-platform gotcha today with the perl POSIX::strftime function. I was after the seconds since the epoch for a given date and had called:

1    # ... determine second, minute, hour, day, month, year
2    my $epoch_seconds = strftime('%s', $second, $minute, $hour
3                                , $day, $month - 1, $year - 1900 );

Which of course works fine under *nix - but it turns out (a bit obviously with hindsight) that the windows strftime does not support '%s'. Unfortunately this does not result in any sort of error - rather you just end up with the literal string '%s'

Time::Local to the rescue - this module implements the reverse of the builtin localtime and gmtime functions.

Here's a version of the above code which works cross-platform:

1    use Time::Local;
2    # ... determine second, minute, hour, day, month, year
3    my $epoch_seconds = timelocal( $second, $minute, $hour
4                                 , $day, $month - 1, $year);

As an aside - the module extends the interpretation of the year to be more user friendly (so you can use fully qualified years, the familiar '$year - 1900' or two-digit years with a rolling window). It's probably better to stick with the fully qualified year - particularly when using old dates as it's possible that '$year - 1900' will end up in a sliding window and drop you in the wrong century.

(For other approaches read `perldoc -q epoch`)

Comment by: Bradley Dean at Thu, 22 Jan 2009 14:18:27 (GMT)

So why isn't this problem more apparent? When I first started looking around for the causes I couldn't really see much. One reason is probably that some developers 'just know' that their strftime has '%s' (not to mention that their time is based on the epoch) while others have never run into the epoch so would never call on '%s'.

The perl strftime documentation refers the reader to the local strftime man page for conversion specifiers, while others (I've looked at python, ruby and php) give a list of available conversion specifiers. I've checked time.strftime in python and it support '%s' (but it's undocumented) which I suspect will be true for most other languages as they'll be wrapping the system strftime rather than reimplimenting it.

Over the years I've found that I've wanted to exclude a user from top (most often 'root' on a fairly quiet system so I can easily see what other user processes are up to even though they're being quiet).

I've written a patch for the top in procps which adds j/J command line and run-time options which essentially mirror the u/U filters.

I've sent this in to the procps maintainers just in case they want to include it in the trunk.

The patch is here: procps-3.2.7-exclude-user-patch.

The full source bundle is here: procps-3.2.7-exclude-user.tgz.

While falling asleep at the keyboard I thought I'd leave a process running overnight to process a bunch of my old files, only to find a series of errors popping up telling me that there were unexpected characters in the filenames.

A quick look revealed that I had ended up with some ISO-8859-1 (ie Latin-1) character encodings within filenames, which was all very well before I switched to UTF-8!.

Contemplating possible temporary resetting of my locale (but what about my newer files that include UTF-8 encodings?) or some sort of simple filter that dropped characters I chanced upon Unicode Tools and a neat list of tools available in various languages.

In this case perl's Encode came to the rescue with this rather short and nicely effective script:

 1  #!/usr/bin/env perl
 2
 3  #
 4  # For each filepath given on stdin, convert the filename from
 5  # iso-8859-1 to utf-8 (or try to!)
 6  #
 7
 8  use strict;
 9  use warnings;
10  use Encode;
11
12  while ( my $orig_filepath = <STDIN> ) {
13    if ( $orig_filepath =~ /[\xA0-\xFF]/ ) {
14      chomp $orig_filepath;
15      my $new_filepath = $orig_filepath;
16      Encode::from_to($new_filepath, 'iso-8859-1', 'utf-8')
17        or die $!;
18      print "$orig_filepath ---> $new_filepath\n";
19      rename $orig_filepath, $new_filepath;
20    }
21  }

The check for characters between 0xA0 and 0xFF is far from perfect for general use (indeed differenciating between the various ISO-8859-*'s would be a bit of a nightmare) but as I knew my encodings were limited to ISO-8859-1 I got away with a bit of a shortcut there.

Comment by: Bradley Dean at Sat, 1 Nov 2008 09:44:11 (BST)

Of course being somewhat late at night I missed the obvious blunder in the logic - there are characters in the range 0xA0-0xFF in UTF-8 encoded strings so rerunning can cause a bit of a mess.

I'm still thinking about this - the UTF-8 multibyte marker byte is in the range 0xC0 to 0xFD. These are all legal ISO-8859-1 character codes so I think I need to draw some sets out on a piece of paper and find the mutually exclusive bits.

Comment by: Bradley Dean at Sun, 2 Nov 2008 00:33:40 (BST)

Given that UTF-8 character bytes for non-ASCII characters fall within the range 0x80 to 0xFD, and given that this overlaps with the character bytes used by ISO-8859-1, there's not a way to differenciate a high-byte string just by looking at individual bytes.

In my scenario (text will either be plain ASCII, ISO-8859-1 or UTF-8) the best test seems to be to try and interpret the string as UTF-8. If errors occur it's going to be ISO-8859-1, if not it's going to be UTF-8.

I've just written Die::Retry - a perl module which makes is easy to write a concise exception retry loop.

The module is only useful if any exception can be silently caught and the code in question re-run, but in that case it makes it very easy to write the loop in question:

1    use Die::Retry qw( retry );
2    my $retval = retry( sub { some_func_with_exceptions($param1, $param2) }
3                            , times => 3
4                            , delay => 0
5                            );

In that example the code ref:

1  sub { some_func_with_exceptions($param1, $param2) }</pre>

is called up to three times if exceptions are thrown. There is no delay between retries. Finally - if a successful call is made to the code ref $retval will get the (scalar context) return value of the code ref call.

I'm submitting this to CPAN and have already set it up on GitHub and Ohloh.

Comment by: Bradley Dean at Mon, 20 Oct 2008 22:56:49 (BST)

I've just found Attempt which does exactly what I coded for Die::Retry - in fact both the code and the syntax are practically identical. So dies Die::Retry.

While travelling through Istanbul on the train I happened to notice that their 'make this seat avaialble to folk in need' sticker held a certain resemblance to a Blur album cover:

Albeit without the "blur" label! :)

After what seems like a bit of an eternity of battling with obstructively gui-driven and complex desktop configuration (such as with gnome and kde), or just plain awful configuration and usage (such as ms windows) I've finally blown the dust off my old Fvwm .fvwmrc file and have re-entered my desktop utopia once again:

Of course I'm back to configuring most things by hand again (wireless, mounting usb drives, X) - but that has always held a certain perverse enjoyment for me anyway!

Here's a sort of long-running private joke between some of my friends and I...

Many years ago we were setting up a server monitor and it was decided that we'd come up with an executive summary style - that being a simple thumbs-up:

Or a thumbs-down:

Somewhere along the way the system developed a bit of an attitude and somehow the thumbs-up became a little bit more like....

Don't really know why I've posted that but a recent conversation lead to re-building those simple images and I thought I'd put them up in lights. :)