<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to you under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->

Short Story:
**************************************************************************
 1. Create top level directory structure - See APPENDIX 1.
 2. Checkout Derby to ${DERBYDIR} - preferably NOT the very latest version
    since we want to use ${TOOLDIR}/updateDerby to create version info for reports.
 2. Copy and modify env.sh, testenv.sh from tools/testing/reporting/TEMPLATES/ to ${DERBYDIR}.
    Copy and modify index.html, index.shtml from tools/testing/reporting/TEMPLATES/ to ${DERBYDIR}/../
 3. ${TOOLDIR}/initDERBYDIR
 4. ${TOOLDIR}/updateDerby [revision]
 5. ${TOOLDIR}/buildDerby "[MKBUILDZIP]" [antpropertyFile]
 6. Create ${DERBYDIR}/testing/builds/baseline.size from ${DERBYDIR}/testing/builds/<revision>.size
 7. Edit ${DERBYDIR}/testing/testhostlist
 8. ${TOOLDIR}/testDerby ${DERBYDIR}/testing/testhostlist
 8. touch ${DERBYDIR}/testing/testlog/${PLATFORM}/externallyVisible for platforms that should be 
    visible in public reports.
 9. ${TOOLDIR}/createBaselineFiles <revision>
10. ${TOOLDIR}/mkAllDerbyTestInfo
    Optional: ${TOOLDIR}/mkAllDerbyTestInfo.public
    Optional: ${TOOLDIR}/mkAllPlotDataAndPlots [INIT] # Use INIT when new baselines created.

After having established the sandbox and report structure you will normally do
4., 5., 8. and 10. 
${TOOLDIR}/updateBuildTestDerby  "[MKBUILDZIP]" "[revision]" [antpropertyfile] executes 4, 5 and 8.


Long Story
****************************************************************************
The scripts assume Derby is checked out to
a directory ${DERBYDIR} 
such that we will have e.g.
 DERBYDIR=${HOME}/Apache/Derby
 BRANCH_DIR=trunk
 derby_source=${DERBYDIR}/${BRANCH_DIR}
or
 DERBYDIR=${HOME}/Apache/Derby-10.1
 BRANCH_DIR=10.1
 derby_source=${DERBYDIR}/${BRANCH_DIR}


The directory structure I have below ${DERBYDIR} is shown in APPENDIX 1..


In ${DERBYDIR}/ I have env.sh and testenv.sh which is sourced to 
set up the environment for update and build(env.sh) and testing(testenv.sh).
Templates which can be copied to ${DERBYDIR}/ and modified to suite your
environment are found in ${derby_source}/tools/testing/reporting/TEMPLATES/.


The scripts are in TOOLDIR=${DERBYDIR}/${BRANCH_DIR}/tools/testing/reporting/scripts
Templates for env.sh and testenv.sh are in 
   TEMPLATEDIR=${DERBYDIR}/${BRANCH_DIR}/tools/testing/reporting/TEMPLATES


Main scripts
************

1. ${TOOLDIR}/updateDerby [revision] (Do '. ./env.sh' first)
   ==================================
   cd's to 'derby_source=${DERBYDIR}/${BRANCH_DIR}' as defined
   in env.sh.
   Does 'svn update'.
   Creates/adds to testing/revisionlist. 
   Creates testing/UpdateInfo/<revision>.txt
   Updates status files(updateOK, update.out) in UpdateLog.
   Creates directory UpdateLog/<revision>/ with status info.
   used in test reports.


2. ${TOOLDIR}/buildDerby "[MKBUILDZIP]" "[antpropertyFile]" (Do '. ./env.sh' first)
   =================================
   cd's to 'derby_source=${DERBYDIR}/${BRANCH_DIR}' as defined
   in env.sh.
   If UpdateLog/updateOK was not created because svn update in 1.
   was not done, build is not attempted.
   
   If build is successful BuildLog/buildOK is created. Creates
   BuildLog/<revision>/ with info used in test reports.

   After the first successful build you should copy ${DERBYDIR}/testing/builds/<revision>.size
   to ${DERBYDIR}/testing/builds/baseline.size and edit it to the format for the 
   baseline.size file shown in this example: # FIXME: create a script...
# 474376
# My.first.build
derby.jar 2257078
derbynet.jar 186619
derbyclient.jar 375758
derbytools.jar 157327
derbyrun.jar 11771
derbyTesting.jar 5563839

   Or you might want to use data for some other build...


3. ${TOOLDIR}/testDerby ${HOSTLISTFILE} (Do . ./testenv.sh first)
   ====================================
   Will exit if UpdateLog/updateOK is not found.
   ${HOSTLISTFILE} (Usually ${DERBYDIR}/testing/testhostlist) must 
   contain the list of machines where tests should be run.
   Assumes ${DERBYDIR} is accessible on all hosts.
   Uses rsh (or ssh if you choose so) to connect to the hosts
   and run "${TOOLDIR}/startDerbyTests ${DERBYDIR}.
   If you just want to run the tests on the build host, remove 
   the loop over machines and run startDerbyTests locally.
   
   ${TOOLDIR}/startDerbyTests executes 
   ${TOOLDIR}/runAllTests using ${DERBYDIR}/testing/testlist which
   lists the Derby testsuites to be run (one testsuite name per line).
   
   Tests are executed in TESTEXECUTIONDIR: e.g.
   TESTEXECUTIONDIR="/export/home/tmp/${USER}/testingDerby<platform>/"
   defined in testenv.sh.

   Test results are reported in ${DERBYDIR}/testing/testlog/<platform>/. 
   ${DERBYDIR}/testing/testlog/ MUST exist.
   A zip file containing the ${DERBYDIR}/testing/testlog/<platform>/ is # FIXME FIXME
   also included in ${DERBYDIR}/testing/testlog/<platform>/. Files
   matching exceptions listed in ${DERBYDIR}/testing/ExcludeFromZip
   are NOT included. I use the following exception:
   
      **.dat
      **.ctrl
      **Tester*.out


 ${TOOLDIR}/updateBuildTestDerby  "[MKBUILDZIP]" "[revision]" [antpropertyfile] 
 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 executes 1, 2 and 3.


4. Creating web presentations
   ==========================

   Important: cd to ${DERBYDIR} and do '. ./env.sh'.

   Execute '${TOOLDIR}/createBaselineFiles <revision>' to
   create a baseline for testsuite execution times for all platforms.
   Or '${TOOLDIR}/createBaselineFilesOnePlatform <revision> <platform>'
   to create a baseline for one specific platform. (<platform>/Directory name as found in
   ${DERBYDIR}/testing/testlog/).

   To make data for a platform publicly visible do
   'touch ${DERBYDIR}/testing/testlog/${PLATFORM}/externallyVisible'.

   The web pages are created by the scripts (check for parameters!)
   ${TOOLDIR}/testSummariesDerby <revision> [details] :
          builds a page for platforms/testsuites.
   ${TOOLDIR}/mkAllDerbyTestInfo
   ${TOOLDIR}/mkAllDerbyTestInfo.public :
          builds pages for current, previous and 2 newest numbered revisions :
                                 ${TOOLDIR}/testSummariesDerby ${LATESTREVISION}
                                 ${TOOLDIR}/testSummariesDerby ${PREVIOUSREVISION}
                                 ${TOOLDIR}/testSummariesDerby current
                                 ${TOOLDIR}/testSummariesDerby previous
          builds pages for revisions/testsuites :
                                 (${TOOLDIR}/testHistoriesDerby)
          builds pages for platforms/revisions :
                                 (${TOOLDIR}/testPlatformsDerby)
          builds top level presentation page :
                                 (${TOOLDIR}/mkDerbyTestIndex)

   'mkAllDerbyTestInfo' and 'mkAllDerbyTestInfo.public' creates index/top level pages for a 
   limited number of tests over revisions. To create a complete index over *all*
   tests where test results have been collected do
   '${TOOLDIR}/mkDerbyTestIndex_all details' and '${TOOLDIR}/mkDerbyTestIndex_all'.
   The index/top level pages contain a link to these labeled "See full history here!".


   With the parameter 'details', pages are created in ${DERBYDIR}/testing/Full/,
   without the parameter 'details', pages are created in ${DERBYDIR}/testing/Limited/.


4.1 Creating test suite duration plots
    ++++++++++++++++++++++++++++++++++
    
   Duration plots are only visible on "Full" pages. Do
   '${TOOLDIR}/mkAllPlotDataAndPlots [INIT]', using 'INIT' the first time and
   after baseline files have been modified.
   NB. Requires gnuplot!
   

4.2 Adding new platforms
    ++++++++++++++++++++
   Assuming test runs have been completed for the new platforms:
  
4.2.1 Create baseline files
      ---------------------
   'cd ${DERBYDIR};. ./env.sh;. ./testenv.sh'.
   For each of the newly added platforms do
   '${TOOLDIR}/createBaselineFilesOnePlatform ${REVISION} ${PLATFORM}'.
   Remember to do '${TOOLDIR}/mkAllPlotDataAndPlots INIT' to reestablish 
   all plots using the new baseline.

4.2.2 Present results on "Limited" pages
      --------------------------
   To show results for the newly added platforms in "Limited" pages create
   the file externallyVisible in 
   ${DERBYDIR}/testing/testlog/${PLATFORM}/.
   E.g. 'touch ${DERBYDIR}/testing/testlog/${PLATFORM}/externallyVisible'.


4.3 Adding revision specific Notes to index pages
    +++++++++++++++++++++++++++++++++++++++++++++

    If you want to display notes on a revision add a <revision number>.txt file
    to ${DERBYDIR}/testing/Notes/.
    HTML formatting is accepted.
    The note is also displayed in the 'testSummary-<revision number>.html' file.


==============================

   OBSERVE: All commands sending mail for update, build and test status are 
   commented out and must be explicitly enabled!


-----------------------------------------------------------------------------
#####################
   OBSERVE: All commands using scp or ssh to copy or manipulate information
   on the external publishing web host are commented out and must be explicitly
    enabled!

   FIXME: Will be removed alltogether: Use e.g. 'rsync' instead for publishing.

#####################


-----------------------------------------------------------------------------

APPENDIX 1.
==========
${DERBYDIR} structure:

BuildLog/
    ...
    474376/
    ...
    474695/
        buildDetails
        buildInfo
        buildInfo.err
        info
        localChanges
        sysinfo.err
        sysinfo.txt
    buildOK
    currentSysInfo.txt
    lastBuildInfo

FailReports/
    ...
    474376.html
    ...
    474695.html

Failures -> trunk/tools/testing/reporting/Failures

TestRunLogs/
    hostA.err
    hostA.log
    ....
    hostN.err
    nostN.log

UpdateInfo/
    ...
    474376.txt
    ...
    474695.txt
    LatestUpdateRevision.txt

UpdateLog/
    ...
    474376/
    ...
    474695/
        UpdateTime
        status-u.err
        status-u.out
        svn--version
        update.err
        update.out

    update.out
    updateOK

builds/
    ...
    474376.size
    ...
    474695.size
    baseline.size

env.sh
summary-full.txt
summary.txt
testenv.sh

testing/
    ExcludeFromZip
    Full/
    Limited/
    Notes/
    ReadyToStartOnCygWin
    UpdateInfo -> ../UpdateInfo
    builds -> ../builds
    revisionlist
    showtestlist -> testlist
    testhostlist
    testlist

    testlog/
        ...
        Linux-2.6.9-34.ELsmp_x86_64-x86_64/
        SunOS-5.10_i86pc-i386/
        SunOS-5.10_sun4u-sparc/
            ...
            474664.zip
            474695-cpu_mem
            474695-system
            474695.csv
            474695.javainfo
            474695.txt
            ...
            baseline.csv
            current.csv
            externallyVisible
            previous.csv
            suiteStart.txt

        ...

trunk/ # or 10.1 or ....
    ...
    ...