Monday 22nd May, 2006 00:10:41 AEST

Transforming between ASCII and EBCDIC

The first time you come across EBCDIC data in an ASCII based environment (or vice versa) things can become a tad confusing - fotunately there's an easy way to convert the data back and forth.

EBCDIC and ASCII are base character encodings - almost all current operating systems use one or the other of these (most using ASCII).

Definitions from Wikipedia

WikiPedia:ASCII :

ASCII (American Standard Code for Information Interchange), generally pronounced [æski], is a character encoding based on the English alphabet. ASCII codes represent text in computers, communications equipment, and other devices that work with text. Most modern character encodings have a historical basis in ASCII.

WikiPedia:EBCDIC :

EBCDIC (Extended Binary Coded Decimal Interchange Code) is an 8-bit character encoding (code page) used on IBM mainframe operating systems, like z/OS, OS/390, VM and VSE, as well as IBM minicomputer operating systems like OS/400 and i5/OS. It is also employed on various non-IBM platforms such as Fujitsu-Siemens' BS2000/OSD, HP MPE/iX, and Unisys MCP. It descended from punched cards and the corresponding six bit binary-coded decimal code that most of IBM's computer peripherals of the late 1950s and early 1960s used.

To transform data between EBCDIC and ASCII the UNIX utility '''dd''' comes to the rescue - better yet, it comes to the rescue in a very simple way.

ASCII to EBCDIC

1  $ cat ascii_data_file | dd conv=ebcdic

EBCDIC to ASCII

1  $ cat ebcdic_data_file | dd conv=ascii

Posted by Bradley Dean | Permalink | Categories: Travel, Programming

Saturday 20th May, 2006 01:31:33 AEST

Perl Programming with Testing

Until I've come up with something to add, for the moment the best way I can think to describe testing with perl is to reference several excelent sources of documentation on the subject:

  • Test::Harness - Run Perl standard test scripts with statistics
  • Test::Simple - Basic utilities for writing tests
  • Test::More - yet another framework for writing test scripts
  • prove - A command-line tool for running tests against Test::Harness

Those provide an excellent basis for constructing and running tests in perl.

Once the processes in those documents and modules have become familiar there are a number of excellent tools that can be used to make testing even more powerful:


Posted by Bradley Dean | Permalink | Categories: Perl, Programming

Saturday 20th May, 2008 00:53:53 AEST

Starting a Catalyst application

Overview

Early experiences with using the Catalyst Web Framework tend to suggest that there are a few common starting points when deploying a new Catalyst application.

These steps take the 'complete' application harness (which is essentially a completely working but completely empty web application) and add a few features to get a little content and functionality into the application.

To begin with it's worth noting that most of documentatin for Catalyst is in the POD - so you can do a lot worse then point your web browser at Catalyst when you're looking for help. The next two resources I've found most useful are the mailing list and the Developer Community Site.

Another invaluable starting point is the Catalyst Tutorial.

Once an application has been created it's time to start adding functionality, so without further ado:

catalyst.pl TestApplication

Using Template Toolkit for the primary View

Often the Template Toolkit will be all you need to manage the View output of an application.

The easiest way to start using Template Toolkit as the default view is to create a view using the Template Toolkit View helper:

./script/testapplication_create.pl view TT TT

Once this is done, the view is most easily accessed by using the DefaultEnd Plugin which will direct all responses without content in the body to the first View. This is done by adding DefaultEnd to the use Catalyst part of lib/TestApplication.pm

use Catalyst qw/ -Debug
                 ConfigLoader
                 Static::Simple
                 DefaultEnd
               /;

A few notes on use of the Template Toolkit view:

  1. A more complex template skeleton site can be automatically generated using the Catalyst::Helper::View::TTSite plugin: ./script/testapplication_create.pl view TT TTSite
  2. If another View is added to the application all is not lost - the DefaultEnd plugin has a view configuration directive:
    # In the YAML configuration file: testapplication.yml
    ---
    name: TestApplication
    view: TT
    
    # OR
    
    # In the application module: lib/TestApplication.pm
    __PACKAGE__->config( name => 'TestApplication',
                         view => 'TT',
                       );
    
    

Changing the default page to be a 401: Not Found

Despite ''looking nice'' returning a '''200 Ok''' for any page requested from a website is very poor form causing all sorts of problems - for instance when spiders start walking through your site. As the default behaviour of the Catalyst base application is to do just that, this should be fixed: in lib/TestApplication/Controller/Root.pm:
#
# Output a friendly welcome message
#
sub default : Private {
    my ( $self, $c ) = @_;

    # If we reach here, the correct reponse is a 404
    $c->response->status(404);

    # Hello World
    $c->response->body( $c->welcome_message );
}
Of course removing the default Catalyst welcome page is also a good idea but can be done later when you get around to putting content into the site.

Adding a wrapper for the View

Because I didn't use the TTSite helper in this example I'll manually add a Template Toolkit wrapper to make building the rest of the pages a little easier. First - add some configuration to the main application (lib/TestApplicatoin.pm):
__PACKAGE__->config( name => 'TestApplication',
                     view => 'TT',
                     'View::TT' => {
                        INCLUDE_PATH => [
                          __PACKAGE__->path_to('templates'),
                          ],
                        WRAPPER => 'site/wrapper.tt',
                        }
                   );
This sets up a template include path in a templates directory in the root application directory. In addition a wrapper is defined. A Template Toolkit wrapper is a template which is wrapped around all content rendered by the library. When writing a wrapper the content of the page will be inserted where the [% content %] directive is placed. An example wrapper is as follows:
[% DEFAULT title = c.config.name %]


  [% title %]


[% title %]

[% content %]

Adding a root index page

Having started the server responding with 404 Not Found perhaps it would be nice to see at least one page that wasn't an error page. To add an index/root/welcome page add a private method called ''index'' to the Root controller:
sub index : Private {
    my ( $self, $c ) = @_;

    # Simple index
    $c->{stash}->{title} = "Index";
    $c->{stash}->{text} = qq{

Welcome to the application - nothing to see here yet

}; $c->{stash}->{template} = "text.tt"; }
In this case the ''text.tt'' template is very simple:
[% text %]
For further information on the automagically called private methods of controllers see Catalyst::Manual::Intro.

Recap...

By this point we have a base catalyst application with the following additions: - Template Toolkit is being used to manage rendering for Views - Changed the default behaviour for the application to return 404 Not Found instead of 200 Ok - Added a wrapper to the Template Toolkit to facilitate templated content - Added an index page This is an excellent starting point to implementing actual functionality with a minimum of effort so could well be a good place to stop this article, but because the MVC framework is all about web applications with backend databases it would be remiss of me not to include a reference to a data source (or Model, as data sources are called in the MVC world).

Adding a SQLite Model

The available helper libraries for Catalyst include several ways to easily incorporate SQL databases - including Class::DBI and DBIx::Class. In this example I will use Class::DBI. In order to properly use data modelling it should not be a requirement that accessing the model is done through the catalyst application. By defining a Class::DBI library independant of the catalyst application and then referring to that library this can be done. If no library is defined for the database yet you may be able to use the automatic database interrogation done for you by Class::Loader. I'm using [http://www.sqlite.org/ SQLite] because it's very simple to do so - if you haven't seen SQLite before go have a look. :) To start with - create an SQLite database (in this case in a {{db}} subdirectory):
BEGIN TRANSACTION;
CREATE TABLE foo (
    bar VARCHAR(132)
);
INSERT INTO "foo" VALUES('sdfasd');
INSERT INTO "foo" VALUES('sdfasd');
INSERT INTO "foo" VALUES('sdfasd');
INSERT INTO "foo" VALUES('sdfasd');
INSERT INTO "foo" VALUES('sdfasd');
COMMIT;
Then add the model:
$ ./script/testapplication_create.pl model AppDB CDBI dbi:SQLite:/path/to/TestApplication/db/db.sqlite 
Once this is done automagically it's helpful for portability to kill that hardwired path to the database - modify the dsn in the created AppDB.pm file from:
    dsn           => 'dbi:SQLite:/path/to/db.sqlite',
to:
    dsn           => 'dbi:SQLite:' . 
                     TestApplication->path_to('db') .
                     '/db.sqlite',
And that's it... No really... To access the Model inside the application use the model method:
my $foo_model = $c->model('AppDB::Foo');
my @foo_rows = $foo_model->retrieve_all();

And all is well...

And there we have it - a dully functioning web application (with no pages or functoins to speak of) but incorporating a fully functioning SQL database abstracted through a data model and using a powerful templating language to minimise data rendering complexity. On top of that you have a full MVC application ''out-of-the-box''. At this point it's probably time to start adding some useful functionality to the application, have fun with that.

Posted by Bradley Dean | Permalink | Categories: Perl, Programming

Friday 19th May, 2006 21:46:19 AEST

Which perl module files are being used?

And now for a super-short article with a fast answer to a problem - on the other hand, it's a problem I really needed a quick answer to the other day that I couldn't find. So here's the article:

Question: Which perl module files are being used during the running of a script?

This can be very useful because it's sometimes very helpful to know which of the installed versions of a module are being used - in my case I needed to build a set of libraries to deploy onto a server on which I could not easily build libraries. The problem was finding out when my script was quietly grabbing a module from the core perl installation instead of my library bundle.

A couple of interesting discussions on PerlMonks on the matter:

  1. Which Library Am I Using?
  2. look which and from where modules were included

There were a couple of different approaches discussed, the most complex of which were ''re-following'' the @INC array to try and find which library ''would'' be used. The problem with that approach is that it's a guess about what will be used rather than a report of what was used.

It turns out there's a very very simple way...

Answer

One of the Perl predefined variables is %INC (not to be confused with the library search path @INC).

As per the perldoc:

  %INC    The hash %INC contains entries for each filename included via the "do", "require",
          or "use" operators.  The key is the filename you specified (with module names con-
          verted to pathnames), and the value is the location of the file found.  The
          "require" operator uses this hash to determine whether a particular file has
          already been included.

          If the file was loaded via a hook (e.g. a subroutine reference, see "require" in
          perlfunc for a description of these hooks), this hook is by default inserted into
          %INC in place of a filename.  Note, however, that the hook may have set the %INC
          entry by itself to provide some more specific info.

So to get a run-time report of what modules are in use, and where the source files were, just print out %INC:

1  use Data::Dumper;
2  print Data::Dumper:;Dumper(\%INC);

For example:

 1  $ perl -MData::Dumper -MEnglish -MCGI -e 'print Data::Dumper::Dumper(\%INC)'
 2  $VAR1 = {
 3            'warnings/register.pm' => '/usr/lib/perl5/5.8.6/warnings/register.pm',
 4            'bytes.pm' => '/usr/lib/perl5/5.8.6/bytes.pm',
 5            'Carp.pm' => '/usr/lib/perl5/5.8.6/Carp.pm',
 6            'XSLoader.pm' => '/usr/lib/perl5/5.8.6/i386-linux-thread-multi/XSLoader.pm',
 7            'English.pm' => '/usr/lib/perl5/5.8.6/English.pm',
 8            'Exporter/Heavy.pm' => '/usr/lib/perl5/5.8.6/Exporter/Heavy.pm',
 9            'vars.pm' => '/usr/lib/perl5/5.8.6/vars.pm',
10            'strict.pm' => '/usr/lib/perl5/5.8.6/strict.pm',
11            'Exporter.pm' => '/usr/lib/perl5/5.8.6/Exporter.pm',
12            'constant.pm' => '/usr/lib/perl5/5.8.6/constant.pm',
13            'warnings.pm' => '/usr/lib/perl5/5.8.6/warnings.pm',
14            'CGI/Util.pm' => '/usr/lib/perl5/5.8.6/CGI/Util.pm',
15            'overload.pm' => '/usr/lib/perl5/5.8.6/overload.pm',
16            'CGI.pm' => '/usr/lib/perl5/5.8.6/CGI.pm',
17            'Data/Dumper.pm' => '/usr/lib/perl5/5.8.6/i386-linux-thread-multi/Data/Dumper.pm'
18          };

Here's the perldoc on PerlDoc:perlvar.

A note on a gotcha

Of course, there must be a gotch...

Modifying %INC is a common way to trick Perl into believing that a module has already been loaded (for instance when using something like Test::MockObject) but when that happens the hash value set is usually not a path to a file. That said - this method is not going to be all that reliable if used in an environment in which munging of %INC is going on.


Posted by Bradley Dean | Permalink | Categories: Perl, Programming