What is Test::Smoke?

Test::Smoke is the symbolic name for set of scripts and modules that try to run the perl core tests on as many configurations as possible and combine the results into an easy to read report.

The basic cycle looks like:

    fetch_sourcetree
    foreach  ( @configurations ) {
        make distclean
        ./Configure 
        make
        make test (and log failures)
    }
    create_report()
    mail_report()

Why is Test::Smoke?

The reports from Test::Smoke give a good picture of the current state of Perl development. We try to get as many of the supported platforms as possible.

Where is Test::Smoke?

On CPAN and http://search.cpan.org/dist/Test-Smoke

What are these configuration files about?

Test::Smoke uses two types of configuration files:

smoke-configuration-file
This is a file that uses a _config suffix and contains settings for the smokeperl.pl program and is created by configsmoke.pl.

This is basically a Data::Dumper::Dump() of a hash.

build-configurations-file
This file contains the information needed to assemble the list of perl configurations that will be used to build and test perl.

The file is devided into sections separated by a line that starts with an equals-sign (=). Each line in a section is combined with all lines from all the other sections. The default build-configurations file perlcurrent.cfg constist of four (4) sections:

section 1 one (1) (blank) line
section 2 two (2) lines
section 3 seven (7) lines
section 4 two (2) lines (and a policy indicator line)

This file wil yield 1 * 2 * 7 * 2 = 28 configurations. Test::Smoke will build perl in all of these 28 configurations and run the core test suite one (1), two (2) or three (3) times for each of those perls:

How do I create different smoke-configurations?

By specifying a diffent prefix to configsmoke.pl:

     perl configsmoke.pl -p mysmoke

This will create the smokeconfiguration mysmoke_config and a small shellscript mysmoke.sh (or mysmoke.cmd if you are on Windows).

There are three (3) special prefixes: smoke55x, smoke562 and smoke58x. These will set the default answers for configsmoke.pl to be appropriate for smoking the maintenance branches for these versions of perl.

How do I include copy-on-write testing in my smokes?

TIMTOWTDO :-)

Most of this works for flags that need to be added to ccflags in general.

-Accflags='-DPERLL_COPY_ON_WRITE'
The fastest way to do it, is to add a new section to your buildconfiguration file, like this:
    =
    # Add a new two line section for -DPERL_COPY_ON_WRITE
    -Accflags='-DPERL_COPY_ON_WRITE'
    =

That will always work, but IMO looks a bit ``loud'' in the reports.

NOTE: This is the only way to smoke COW on MSWin32!

Default Policy.sh >
Since maintenence version 1.18.10, the default Policy.sh is generated from the policy-target sections in the buildconfigurations file. It is now possible to just add this three line section to your buildconfiguration file:
    =
    /-DPERL_COPY_ON_WRITE/
    -DPERL_COPY_ON_WRITE
    =
Your own Policy.sh >
Some people use the posibility to handle site specific build environments from <builddir/../Policy.sh >>. This excludes the use of the default Policy.sh. You will need to add the -DPERL_COPY_ON_WRITE flag to the ccflags='' line in your Policy.sh. After that you must add the policy-target section, as above, to your buildconfiguration file.

What are all the scripts in the smoke suite for?

smokeperl.pl
This is a wrapper script that goes through all the steps in the smoke process.
synctree.pl
Small wrapper around Test::Smoke::Syncer that can use the settings from the smoke-configuration file. It will bring the source-tree in sync.
patchtree.pl
Small wrapper around Test::Smoke::Patcher that can use the settings from the smoke-configuration file. It will apply the patches that are specified in the patches-to-be-applied file.
runsmoke.pl
Small wrapper around Test::Smoke::run_smoke() that can use the settings from the smoke-configuration file. It runs the smoke cycle for all configurations and is a replacement for the now removed mktest.pl.
mailrpt.pl
Small wrapper around Test::Smoke::Mailer that can use the settings from the smoke-configuration file. It will check if the report is already generated and do so if not. This is the replacement for the now removed mkovz.pl.
archiverpt.pl
This script copies the report and the logfile to a configurable place.
smokestatus.pl
This script prints a little status report for a smoke confifuration:
     ./smokestatus.pl -c smoke58x
    Checking status for configuration 'smoke58x' (5.8.x)
      Change number 21537 started on Sun Oct 26 06:30:39 2003.
        4 out of 10 configurations finished in 3 hours 6 minutes.
        1 configuration showed failures (F).
        0 failures in the running configuration.
        6 configurations to finish, estimated completion in 4 hours 40 minutes
        Average smoke duration: 46 minutes 42 seconds.
W32Configure.bat
Small wrapper around Test::Smoke::Util::Configure_win32() that can use some of the settings from the smoke-configuration file.

This helps users on Windows to create the < <builddir/win32/smoke.mk >> makefile for a specific configuration in a unixy way.

How can I skip a step in smokeperl.pl?

smokeperl.pl takes arguments to skip some of the steps it takes:

--nofetch Do not sync the source-tree before smoking
--nopatch Do not patch the source-tree (if applicable)
--nomail Do not send the report out by mail
--noarchive Do not archive the report and logfile

smokeperl.pl will always do the equivalent of runsmoke.pl and generate the report.

After running configsmoke.pl you should have a small wrapper-script, that sets up the environment and calls smokeperl.pl. All arguments passed to this script will be passed to smokeperl.pl! Please see smokeperl.pl for details.

How do I smoke my patch?

When preparing a patch for admission to perl5-porters it is always a good idea to test it. Test::Smoke has a mechanism to smoke-test the effects of that patch. Here is a way to set it up:

Set up Test::Smoke
Install Test::Smoke and create a separate directory to keep all your configurations in.
Create a build-configurations file
In this configs directory you then create a build-configurations file. This is usually done from one of default *.cfg files in the Test::Smoke distribution.
Configure Test::Smoke with configsmoke.pl
Now you need to configure Test::Smoke by running configsmoke.pl in your configs directory (optionally specifying a prefix).
     ../smoke/configsmoke.pl -p testpatch

You should be careful with the e-mail options and make sure you do not send the results to either <smokers-reports@perl.org> or <perl5-porters@perl.org>.

Create a patches-to-apply-file
The patches-to-apply-file has a simple format. It has each patch to apply on a single line and each line is semicolon separated with the full path to the patchfile in the first column and optionally the options to pass to patch(1) in the second column. By default -p1 is passed if no -pN option was specified.
     cat mypatch.inc
    ~/p5p/patches/mypatch.diff;-p0
Start the smoke with the --pfile option
     ./testpatch.sh --pfile mypatch.inc
 
You can also configure Test::Smoke to always use a
patches-to-apply-file during F<configsmoke.pl>.

Can I interrupt a smoke run?

Yes, you can kill all the smoke related processes. Each stage is directly logged in <builddir/mktest.out >>.

As of version 1.17.02 you can specify a timeout (if you have alarm())

Can I still generate a report after an interrupted smoke?

Yes, but some information might be missing (the compiler info).

     perl mailrpt.pl -c [configname]

How can I run continues smokes?

On a unixy system you can try this shellscript if you adjust the basedir variable (thank you Johan Vromans for helping with this).

    #! /bin/sh
    
    basedir=/Test-Smoke/smoke
    cd ""
    pidfile=`pwd`/smokeloop.pid
    stpfile=`pwd`/.stopsmoking
    versions="58x current"
    if test -f "" ; then
        echo "We seem to be running as: `cat `"
        exit
    fi
    trap 'trap "" 0 ; rm -f "" ; echo "exiting now" ; exit' 0 1 2 3 15
    checkstop() {
        if test -f "" ; then
            echo "`date`: Stopping on user request"
            rm -f ""
            exit
        fi
    }
    echo  > ""
    while test 1 ; do
        checkstop
        touch ""
        for smokever in  ; do
            nice /smoke
            checkstop
        done
        perl -e 'sleep 3600 if (1+2*rand)*(time-(stat(shift))[9])<600' \
            ""
    done

This script assumes it is located in the directory where the Test::Smoke suite is installed. You can stop execution between smokes for different versions by putting the file .stopsmoking in that same directory.

Where do the reports go?

To the address configured in configsmoke.pl. The default is <smokers-reports@perl.org>.

There are several ways to monitor the reports send in:

Subscribe to the mailinglist
You can subscribe to that list by sending an empty message to <daily-build-reports-subscribe@perl.org>. You do not have to be subscribed to send reports!
The usenet mirror
The mailinglist also has a mirror on usenet: <news:nntp.perl.org/perl.daily-build.reports>

Posts to the newsgroup will also be mirrored on the mailinglist.

The usenet WWW archive
The usenet mirror also has an archive on the web: http://www.nntp.perl.org/group/perl.daily-build.reports

How do I investigate failures?

When investigating a testfailure you need to make sure you have Configured and build perl with the same options as the one with the failures.

     cd <builddir>
     make distclean
     sh Configure -des -Dusedevel [other build options]
     make
     make test-prep
     cd t

Now you can start with a manual run of the failing test:

     ./perl harness -v <pathto/failingtest>
Note for Win32
As Windows lacks a Configure to create the Makefile, Test::Smoke supplies the helper script W32Configure.bat that will create a new Makefile with the right options: <builddir\win32\smoke.mk >>.

After that you can:

    S:\perl-current\win32>nmake -f smoke.mk
    S:\perl-current\win32>nmake -f smoke.mk test-prep
    S:\perl-current\win32>cd ..\t

For -Duseperlio builds (per default) failures can occur in different environments, and you will also need to set these up:

PERLIO=stdio
     PERLIO=stdio ./perl harness -v <pathto/failingtest>

Note that Windows builds (MSVC, Borland or MinGW) also need the ``crlf'' layer:

    S:>(set PERLIO=stdio :crlf) && .\perl harness -v <pathto\failingtest>

Be careful with the environment variables on Windows, they are sticky.

PERLIO=perlio
     PERLIO=perlio ./perl harness -v <pathto/failingtest>

Note that Windows builds (MSVC, Borland or MinGW) also need the ``crlf'' layer:

    S:>(set PERLIO=perlio :crlf) && .\perl harness -v <pathto\failingtest>

Be careful with the environment variables on Windows, they are sticky.

UTF-8 locale
If the failure occurs in the UTF-8 locale test-run the environment setup is a bit different (assuming ``en_US.utf8''):
     PERL_UNICODE= LC_ALL=en_US.utf8 ./perl harness -v <pathto/failingtest>

What's with the ``daily-build'' and ``smokers'' names?

There has been enough discussion on the subject, so lets agree to disagree (it was all before my time).

The case is now that the official name for the mailinglist is daily-build-reports and there is an alias to smokers-reports.

How do I resend a report

Sometimes the report server is down or the report generated an internal error on the report serer. In this case one could resend the report for any given sha (say a1e3603d9bc6dda037cac0f2d3dc1dbf59bdf06f) like this:

perl -MCGI::Util=escape -E 'my $j=do{local $/; <ARGV>}; say "json=" . escape($j)' \
	logs/smokecurrent/jsna1e3603d9bc6dda037cac0f2d3dc1dbf59bdf06f.jsn > tmp_payload
curl -i https://perl5.test-smoke.org/report -XPOST -d@tmp_payload