PAR::Tutorial.3pm

Langue: en

Autres versions - même langue

Version: 2007-12-20 (mandriva - 01/05/08)

Section: 3 (Bibliothèques de fonctions)

Sommaire

NAME

PAR::Tutorial - Cross-Platform Packaging and Deployment with PAR

SYNOPSIS

This is a tutorial on PAR, first appeared at the 7th Perl Conference. The HTML version of this tutorial is available online as <http://aut.dyndns.org/par-tutorial/>.

DESCRIPTION


On Deploying Perl Applications

  % sshnuke.pl 10.2.2.2 -rootpw="Z1ON0101"
  Perl v5.6.1 required--this is only v5.6.0, stopped at sshnuke.pl line 1.
  BEGIN failed--compilation aborted at sshnuke.pl line 1.
•
Q: ``Help! I can't run your program!''
•
A1: Install Perl & "perl -MCPAN -e'install(...)'"
•
How do we know which modules are needed?
•
New versions of CPAN modules may break "sshnuke.pl"
•
A2: Install Perl & "tar zxf my_perllib.tgz"
•
Possibly overwriting existing modules; not cross-platform at all
•
A3: Use the executable generated by "perlcc sshnuke.pl"
•
Impossible to debug; "perlcc" usually does not work anyway

PAR, the Perl Archive Toolkit

•
Do what JAR (Java Archive) does for Perl
•
Aggregates modules, scripts and other files into a Zip file
•
Easy to generate, update and extract
•
Version consistency: solves forward-compatibility problems
•
Developed by community: "par@perl.org"
•
PAR files can be packed into self-contained scripts
•
Automatically scans perl script for dependencies
•
Bundles all necessary 3rd-party modules with it
•
Requires only core Perl to run on the target machine
•
PAR also comes with "pp", the Perl Packager: 
  % pp -o sshnuke.exe sshnuke.pl # stand-alone executable!

Simple Packaging

•
PAR files are just Zip files with modules in it
•
Any Zip tools can generate them: 
  % zip foo.par Hello.pm World.pm        # pack two modules
  % zip -r bar.par lib/          # grab all modules in lib/
•
To load modules from PAR files: 
  use PAR;
  use lib "foo.par";             # the .par part is optional
  use Hello;
•
This also works: 
  use PAR "/home/mylibs/*.par";  # put all of them into @INC
  use Hello;

PAR Loaders

•
Use "par.pl" to run files inside a PAR archive: 
  % par.pl foo.par               # looks for 'main.pl' by default
  % par.pl foo.par test.pl       # runs script/test.pl in foo.par
•
Same thing, with the stand-alone "parl" or "parl.exe": 
  % parl foo.par                 # no perl or PAR.pm needed!
  % parl foo.par test.pl         # ditto
•
The PAR loader can prepend itself to a PAR file:
•
"-b" bundles non-core modules needed by "PAR.pm": 
  % par.pl -b -O./foo.pl foo.par # self-contained script
•
"-B" bundles core modules in addition to "-b": 
  % parl -B -O./foo.exe foo.par  # self-contained binary

Dependency Scanning

•
Recursively scan dependencies with "scandeps.pl": 
  % scandeps.pl sshnuke.pl
  # Legend: [C]ore [X]ternal [S]ubmodule [?]NotOnCPAN
  'Crypt::SSLeay'       => '0', #  X   #
  'Net::HTTP'           => '0', #      #
  'Crypt::SSLeay::X509' => '0', # S    # Crypt::SSLeay
  'Net::HTTP::Methods'  => '0', # S    # Net::HTTP
  'Compress::Zlib'      => '0', #  X   # Net::HTTP::Methods
•
Scan an one-liner, list all involved files: 
  % scandeps.pl -V -e "use Dynaloader;"
  ...
  # auto/DynaLoader/dl_findfile.al [autoload]
  # auto/DynaLoader/extralibs.ld [autoload]
  # auto/File/Glob/Glob.bs [data]
  # auto/File/Glob/Glob.so [shared]
  ...

Perl Packager: "pp"


Perl Packager: pp

•
Combines scanning, zipping and loader-embedding: 
  % pp -o out.exe src.pl         # self-contained .exe
  % out.exe                      # runs anywhere on the same OS
•
Bundle additional modules: 
  % pp -o out.exe -M CGI src.pl  # pack CGI + its dependencies, too
•
Pack one-liners: 
  % pp -o out.exe -e 'print "Hi!"'   # turns one-liner into executable
•
Generate PAR files instead of executables: 
  % pp -p src.pl                 # makes 'source.par'
  % pp -B -p src.pl              # include core modules

How it works

•
Command-line options are almost identical to "perlcc"'s
•
Also supports "gcc"-style long options: 
  % pp --gui --verbose --output=out.exe src.pl
•
Small initial overhead; no runtime overhead
•
Dependencies are POD-stripped before packing
•
Loads modules directly into memory on demand
•
Shared libraries (DLLs) are extracted with File::Temp
•
Works on Perl 5.6.0 or above
•
Tested on Win32 (VC++ and MinGW), FreeBSD, NetBSD, Linux, MacOSX, Cygwin, AIX, Solaris, HP-UX, Tru64...

Aggregating multiple programs

•
A common question: 
  > I have used pp to make several standalone applications which work
  > great, the only problem is that for each executable that I make, I am
  > assuming the parl.exe is somehow bundled into the resulting exe.
•
The obvious workaround: 
  You can ship parl.exe by itself, along with .par files built
  by "pp -p", and run those PAR files by associating them to parl.exe.
•
On platforms that have "ln", there is a better solution: 
  % pp --output=a.out a.pl b.pl  # two scripts in one!
  % ln a.out b.out               # symlink also works
  % ./a.out                      # runs a.pl
  % ./b.out                      # runs b.pl

Cross-platform Packages

•
Of course, there is no cross-platform binary format
•
Pure-perl PAR packages are cross-platform by default
•
However, XS modules are specific to Perl version and platform
•
Multiple versions of a XS module can co-exist in a PAR file
•
Suppose we need "out.par" on both Win32 and Finix: 
  C:\> pp --multiarch --output=out.par src.pl
  ...copy src.pl and out.par to a Finix machine...
  % pp --multiarch --output=out.par src.pl
•
Now it works on both platforms: 
  % parl out.par                 # runs src.pl
  % perl -MPAR=out.par -e '...'  # uses modules inside out.par

The Anatomy of a PAR file

•
Modules can reside in several directories: 
  /                      # casual packaging only
  /lib/                  # standard location
  /arch/                 # for creating from blib/ 
  /i386-freebsd/         # i.e. $Config{archname}
  /5.8.0/                # i.e. Perl version number
  /5.8.0/i386-freebsd/   # combination of the two above
•
Scripts are stored in one of the two locations: 
  /                      # casual packaging only
  /script/               # standard location
•
Shared libraries may be architecture- or perl-version-specific: 
  /shlib/(5.8.0/)?(i386-freebsd/)?
•
PAR files may recursively contain other PAR files: 
  /par/(5.8.0/)?(i386-freebsd/)?

Special files

•
MANIFEST
•
Index of all files inside PAR
•
Can be parsed with "ExtUtils::Manifest"
•
META.yml
•
Dependency, license, runtime options
•
Can be parsed with "YAML"
•
SIGNATURE
•
OpenPGP-signed digital signature
•
Can be parsed and verified with "Module::Signature"

Advantages over perlcc, PerlApp and Perl2exe

•
This is not meant to be a flame
•
All three maintainers have contributed to PAR directly; I'm grateful
•
perlcc
•
``The code generated in this way is not guaranteed to work... Use for production purposes is strongly discouraged.'' (from perldoc perlcc)
•
Guaranteed to not work is more like it
•
PerlApp / Perl2exe
•
Expensive: Need to pay for each upgrade
•
Non-portable: Only available for limited platforms
•
Proprietary: Cannot extend its features or fix bugs
•
Obfuscated: Vendor and black-hats can see your code, but you can't
•
Inflexible: Does not work with existing Perl installations

MANIFEST: Best viewed with Mozilla

•
The URL of "MANIFEST" inside "/home/autrijus/foo.par": 
  jar:file:///home/autrijus/foo.par!/MANIFEST
•
Open it in a Gecko browser (e.g. Netscape 6+) with Javascript enabled:
•
No needed to unzip anything; just click on files to view them

META.yml: Metadata galore

•
Static, machine-readable distribution metadata
•
Supported by "Module::Build", "ExtUtils::MakeMaker", "Module::Install"
•
A typical "pp"-generated "META.yml" looks like this: 
  build_requires: {}
  conflicts: {}
  dist_name: out.par
  distribution_type: par
  dynamic_config: 0
  generated_by: 'Perl Packager version 0.03'
  license: unknown
  par:
    clean: 0
    signature: ''
    verbatim: 0
    version: 0.68
•
The "par:" settings controls its runtime behavior

SIGNATURE: Signing and verifying packages

•
OpenPGP clear-signed manifest with SHA1 digests
•
Supported by "Module::Signature", "CPANPLUS" and "Module::Build"
•
A typical "SIGNATURE" looks like this: 
  -----BEGIN PGP SIGNED MESSAGE-----
  Hash: SHA1
 
  SHA1 8a014cd6d0f6775552a01d1e6354a69eb6826046 AUTHORS
  ...
  -----BEGIN PGP SIGNATURE-----
  ...
  -----END PGP SIGNATURE-----
•
Use "pp" and "cpansign" to work with signatures: 
  % pp -s -o foo.par bar.pl      # make and sign foo.par from bar.pl
  % cpansign -s foo.par  # sign this PAR file
  % cpansign -v foo.par  # verify this PAR file

Perl Servlets with Apache::PAR

•
Framework for self-contained Web applications
•
Similar to Java's ``Web Application Archive'' (WAR) files
•
Works with mod_perl 1.x or 2.x
•
A complete web application inside a ".par" file
•
Apache configuration, static files, Perl modules...
•
Supports Static, Registry and PerlRun handlers
•
Can also load all PARs under a directory
•
One additional special file: "web.conf" 
  Alias /myapp/cgi-perl/ ##PARFILE##/
  <Location /myapp/cgi-perl>
      Options +ExecCGI
      SetHandler perl-script
      PerlHandler Apache::PAR::Registry
  </Location>

Hon Dah, A-par-che!

•
First, make a "hondah.par" from an one-liner: 
  # use the "web.conf" from the previous slide
  % pp -p -o hondah.par -e 'print "Hon Dah!\n"' \
       --add web.conf
  % chmod a+x hondah.par
•
Add this to "httpd.conf", then restart apache: 
  <IfDefine MODPERL2>
  PerlModule Apache2
  </IfDefine>
  PerlAddVar PARInclude /home/autrijus/hondah.par
  PerlModule Apache::PAR
•
Test it out: 
  % GET http://localhost/myapp/cgi-perl/main.pl
  Hon Dah!
•
Instant one-liner web application that works!

On-demand library fetching

•
With LWP installed, your can use remote PAR files: 
  use PAR;
  use lib 'http://aut.dyndns.org/par/DBI-latest.par';
  use DBI;    # always up to date!
•
Modules are now cached under $ENV{PAR_GLOBAL_TEMP}
•
Auto-updates with "LWP::Simple::mirror"
•
Download only if modified
•
Safe for offline use after the first time
•
May use "SIGNATURE" to prevent DNS-spoofing
•
Makes large-scale deployment a breeze
•
Upgrades from a central location
•
No installers needed

Code Obfuscation

•
Also known as source-hiding techniques
•
It is not encryption
•
Offered by PerlApp, Perl2Exe, Stunnix...
•
Usually easy to defeat
•
Take optree dump from memory, feed to "B::Deparse"
•
If you just want to stop a casual "grep", ``deflate'' already works
•
PAR now supports pluggable input filters with "pp -f"
•
Bundled examples: Bleach, PodStrip and PatchContent
•
True encryption using "Crypt::*"
•
Or even _product activation_ over the internet
•
Alternatively, just keep core logic in your server and use RPC

Accessing packed files

•
To get the host archive from a packed program: 
  my $zip = PAR::par_handle($0); # an Archive::Zip object
  my $content = $zip->contents('MANIFEST');
•
Same thing, but with "read_file()": 
  my $content = PAR::read_file('MANIFEST');
•
Loaded PAR files are stored in %PAR::LibCache: 
  use PAR '/home/mylibs/*.par';
  while (my ($filename, $zip) = each %PAR::LibCache) {
      print "[$filename - MANIFEST]\n";
      print $zip->contents('MANIFEST');
  }

Packing GUI applications

•
GUI toolkits often need to link with shared libraries: 
  # search for libncurses under library paths and pack it
  % pp -l ncurses curses_app.pl  # same for Tk, Wx, Gtk, Qt...
•
Use "pp --gui" on Win32 to eliminate the console window: 
  # pack 'src.pl' into a console-less 'out.exe' (Win32 only)
  % pp --gui -o out.exe src.pl
•
``Can't locate Foo/Widget/Bar.pm in @INC''?
•
Some toolkits (notably Tk) autoloads modules without "use" or "require"
•
Hence "pp" and "Module::ScanDeps" may fail to detect them
•
Tk problems mostly fixed by now, but other toolkits may still break
•
You can work around it with "pp -M" or an explicit "require"
•
Or better, send a short test-case to "par@perl.org" so we can fix it

Precompiled CPAN distributions

•
Installing XS extensions from CPAN was difficult
•
Some platforms do not come with a compiler (Win32, MacOSX...)
•
Some headers or libraries may be missing
•
PAR.pm itself used to suffer from both problems
•
...but not anymore --- "Module::Install" to the rescue! 
  # same old Makefile.PL, with a few changes
  use inc::Module::Install;      # was "use ExtUtils::MakeMaker;"
  WriteMakefile( ... );          # same as the original
  check_nmake();                 # make sure the user have nmake
  par_base('AUTRIJUS');          # your CPAN ID or a URL
  fetch_par() unless can_cc();   # use precompiled PAR only if necessary
•
Users will not notice anything, except now it works
•
Of course, you still need to type "make par" and upload the precompiled package
•
PAR users can also install it directly with "parl -i"

Platform-specific Tips

•
Win32 and other icon-savvy platforms
•
Needs 3rd-party tools to add icons to "pp"-generated executables
•
PE Header manipulation in Perl --- volunteers wanted!
•
Linux and other libc-based platforms
•
Try to avoid running "pp" on a bleeding-edge version of the OS
•
Older versions with an earlier libc won't work with new ones
•
Solaris and other zlib-lacking platforms (but not Win32)
•
You need a static-linked "Compress::Zlib" before installing PAR
•
In the future, PAR may depend on "Compress::Zlib::Static" instead
•
Any platform with limited bandwidth or disk space
•
Use UPX to minimize the executable size

Thank you!

•
Additional resources
•
Mailing list: "par@perl.org"
•
Subscribe: Send a blank email to "par-subscribe@perl.org"
•
List archive: <http://nntp.x.perl.org/group/perl.par>
•
PAR::Intro: <http://search.cpan.org/dist/PAR/lib/PAR/Intro.pod>
•
Apache::PAR: <http://search.cpan.org/dist/Apache-PAR/>
•
Module::Install: <http://search.cpan.org/dist/Module-Install/>
•
Any questions?

Bonus Slides: PAR Internals


Overview of PAR.pm's Implementation

•
Here begins the scary part
•
Grues, Dragons and Jabberwocks abound...
•
You are going to learn weird things about Perl internals
•
PAR invokes four areas of Perl arcana:
•
@INC code references
•
On-the-fly source filtering
•
Overriding "DynaLoader::bootstrap()" to handle XS modules
•
Making self-bootstrapping binary executables
•
The first two only works on 5.6 or later
•
DynaLoader and %INC are there since Perl 5 was born
•
PAR currently needs 5.6, but a 5.005 port is possible

Code References in @INC


Code References in @INC

•
On 1999-07-19, Ken Fox submitted a patch to P5P
•
To _enable using remote modules_ by putting hooks in @INC
•
It's accepted to come in Perl 5.6, but undocumented until 5.8
•
Type "perldoc -f require" to read the nitty-gritty details
•
Coderefs in @INC may return a fh, or undef to 'pass': 
  push @INC, sub {
      my ($coderef, $filename) = @_;  # $coderef is \&my_sub
      open my $fh, "wget ftp://example.com/$filename |";
      return $fh;        # using remote modules, indeed!
  };
•
Perl 5.8 let you open a file handle to a string, so we just use that: 
         open my $fh, '<', \($zip->memberNamed($filename)->contents);
         return $fh;
•
But Perl 5.6 does not have that, and I don't want to use temp files...

Source Filtering without Filter::* Modules

•
... Undocumented features to the rescue!
•
It turns out that @INC hooks can return two values
•
The first is still the file handle
•
The second is a code reference for line-by-line source filtering!
•
This is how "Acme::use::strict::with::pride" works: 
  # Force all modules used to use strict and warnings
  open my $fh, "<", $filename or return;
  my @lines = ("use strict; use warnings;\n", "#line 1 \"$full\"\n");
  return ($fh, sub {
      return 0 unless @lines;    
      push @lines, $_; $_ = shift @lines; return length $_;
  });

Source Filtering without Filter::* Modules (cont.)

•
But we don't really have a filehandle for anything
•
Another undocumented feature saves the day!
•
We can actually omit the first return value altogether: 
  # Return all contents line-by-line from the file inside PAR
  my @lines = split(
      /(?<=\n)/,
      $zip->memberNamed($filename)->contents
  );
  return (sub {
      $_ = shift(@lines);
      return length $_;
  });

Overriding DynaLoader::bootstrap

•
XS modules have dynamically loaded libraries
•
They cannot be loaded as part of a zip file, so we extract them out
•
Must intercept DynaLoader's library-finding process
•
Module names are passed to "bootstrap" for XS loading
•
During the process, it calls "dl_findfile" to locate the file
•
So we install pre-hooks around both functions
•
Our "_bootstrap" just checks if the library is in PARs
•
If yes, extract it to a "File::Temp" temp file
•
The file will be automatically cleaned up when the program ends
•
It then pass the arguments to the original "bootstrap"
•
Finally, our "dl_findfile" intercepts known filenames and return it

Anatomy of a Self-Contained PAR executable

•
The par script ($0) itself
•
May be in plain-text or native executable format
•
Any number of embedded files
•
Typically used to bootstrap PAR's various dependencies
•
Each section begins with the magic string ``FILE''
•
Length of filename in pack('N') format and the filename (auto/.../)
•
File length in pack('N') and the file's content (not compressed)
•
One PAR file
•
Just a regular zip file with the magic string "PK\003\004"
•
Ending section
•
A pack('N') number of the total length of FILE and PAR sections
•
Finally, there must be a 8-bytes magic string: "\012PAR.pm\012"

Self-Bootstrapping Tricks

•
All we can expect is a working perl interpreter
•
The self-contained script *must not* use any modules at all
•
But to process PAR files, we need XS modules like Compress::Zlib
•
Answer: bundle all modules + libraries used by PAR.pm
•
That's what the "FILE" section in the previous slide is for
•
Load modules to memory, and write object files to disk
•
Then use a local @INC hook to load them on demand
•
Minimizing the amount of temporary files
•
First, try to load PerlIO::scalar and File::Temp
•
Set up an END hook to unlink all temp files up to this point
•
Load other bundled files, and look in the compressed PAR section
•
This can be much easier with a pure-perl "inflate()"; patches welcome!

Thank you (again)!

•
Any questions, please?

SEE ALSO

<http://www.autrijus.org/par-tutorial/>

<http://www.autrijus.org/par-intro/> (English version)

<http://www.autrijus.org/par-intro.zh/> (Chinese version)

PAR, pp, par.pl, parl

ex::lib::zip, Acme::use::strict::with::pride

App::Packer, Apache::PAR, CPANPLUS, Module::Install

AUTHORS

Audrey Tang <cpan@audreyt.org>

<http://par.perl.org/> is the official PAR website. You can write to the mailing list at <par@perl.org>, or send an empty mail to <par-subscribe@perl.org> to participate in the discussion.

Please submit bug reports to <bug-par@rt.cpan.org>.

Copyright 2003, 2004, 2005, 2006 by Audrey Tang <cpan@audreyt.org>.

This document is free documentation; you can redistribute it and/or modify it under the same terms as Perl itself.

See <http://www.perl.com/perl/misc/Artistic.html>