This file is an attempt to start to document how the reporter.apache.org site is set up.

It is currently very rudimentary.

NOTE: Most of the stats reported on the reporter page come from a running instance of Apache Kibble at https://demo.kibble.apache.org.

Code for Kibble is available at https://kibble.apache.org/docs/downloads.html


Original version:
  Javascript and CSS are Foundation
  http://foundation.zurb.com
  Current version seems to be 5.5.1 (see start of site/css/foundation.css)

NG version:
  jQuery and custom CSS/CoffeeScript compiled into tabs.js using:
    cat coffee/*.coffee | coffee -b --compile --stdio > tabs.js

Also uses Google Loader:
https://developers.google.com/loader/
This is used by site/index.html which loads the visualization API modules: corechart, timeline

The site runs on the host projects-vm3.a.o at present (May 2021)

The puppet config for the VM is stored at:
https://github.com/apache/infrastructure-p6/blob/production/data/nodes/projects-vm3.apache.org.yaml
(private repo, requires GH account linked to ASF)

There are other puppet data files for the host under:
https://github.com/apache/infrastructure-p6/blob/production/modules/projects_pvm_asf/

The ASF-hosted private Git repo is at:
https://gitbox.apache.org/repos/private/infrastructure?p=infrastructure-p6.git

Crontab:

crontab -l -u root (in puppet, part of https://github.com/apache/infrastructure-p6/blob/production/modules/projects_pvm_asf/manifests/init.pp):
# m h  dom mon dow   command

# Puppet Name: rao_releases_ci
45 0,6,12,18 * * * cd /var/www/reporter.apache.org/data/releases && sudo -n -u www-data svn ci -m "updating report releases data" --username projects_role --password `cat /root/.rolepwd` --non-interactive >>/var/log/www-data-root/svnreleases_$(date "+\%Y-\%m").log
# Puppet Name: rao_history_ci
10 4,12,20 * * * cd /var/www/reporter.apache.org/data/history && sudo -n -u www-data svn ci -m "updating report releases data" --username projects_role --password `cat /root/.rolepwd` --non-interactive >>/var/log/www-data-root/svnhistory_$(date "+\%Y-\%m").log
# Puppet Name: pao_json_ci
20 4 * * * cd /var/www/projects.apache.org/site/json && sudo -n -u www-data svn ci -m "updating projects data" --username projects_role --password `cat /root/.rolepwd` --non-interactive >>/var/log/www-data-root/svnjson_$(date "+\%Y-\%m").log

crontab -l -u www-data:
# m h   dom mon dow   command
00 4,12,20 * * * cd /var/www/reporter.apache.org/scripts && ./python3logger.sh parsepmcs.py
00 01 * * * cd /var/www/reporter.apache.org/scripts && ./python3logger.sh mailglomper2.py
00 09 * * *      cd /var/www/reporter.apache.org/scripts && ./python3logger.sh readjira.py
10 00 * * *      cd /var/www/reporter.apache.org/scripts && ./python3logger.sh reportingcycles.py
20 00 * * *      cd /var/www/reporter.apache.org/scripts && ./python3logger.sh pmcdates.py
30 00 * * *      cd /var/www/reporter.apache.org/scripts && ./python3logger.sh bugzillastats.py
50 00 * * *      cd /var/www/reporter.apache.org/scripts && ./python3logger.sh health.py
32  * * * *      cd /var/www/reporter.apache.org/scripts && ./python3logger.sh readchecker.py
*/15 * * * *     pgrep -f "gunicorn3.*wsgi:app" || (cd /var/www/reporter.apache.org/scripts && gunicorn3 -b 127.0.0.1 -w 6 -D wsgi:app && echo "restarting gunicorn")

# ensure that any new data files get picked up by the commit (which must be done by root)
40  * * * *      cd /var/www/reporter.apache.org/scripts && ./svnadd.sh ../data/releases

# Run scandist
@reboot         cd /var/www/reporter.apache.org/scripts && ./scandist.sh
@monthly        cd /var/www/reporter.apache.org/scripts && ./scandist.sh restart

There are additional jobs for projects.a.o which are documented in its file STRUCTURE.txt

Scripts:
- scripts/health.py
  Creates data/health.json

- scripts/parsepmcs.py
  Updates data/pmcs.json and data/projects.json (from Whimsy public data)
  Also updates historic copies (without the last seen timestamp) in data/history

-scripts/pmcdates.py
  Creates data/pmcdates.json from committee_info.json

-scripts/reportingcycles.py
  Creates site/reportingcycles.json from committee_info.json

- scripts/mailglomper2.py
  Updates data/maildata_extended.json from https://lists.apache.org/

- scripts/readchecker.py
  Caches output of https://checker.apache.org/json/ in data/cache/checker.json

- scripts/readjira.py
   For each .json file found under data/JSON (apart from jira_projects.json) it recreates the file.
   Also refreshes data/JIRA/jira_projects.json

- site/addrelease.py
  Updates data/releases/%s.json % committee from form data

- site/getjson.py
  If stale, re-creates data/JIRA/jira_projects.json (from JIRA)
  If stale, re-creates data/JIRA/%s.json % project (from JIRA)
  
- site/jiraversions.py
  Updates data/releases/%s.json % project

Data file consumers:
- chi.py
  data/health.json

- health.py
  data/maildata_extended.json
  data/pmcs.json
  data/projects.json
  data/releases/%s.json % project
  https://whimsy.apache.org/public/committee-info.json

- render.js
  site/reportingcycles.json
  site/getjson.py?only=project
  site/jiraversions.py?project=<pmc>&jiraname=<project>&prepend=<prepend>
  site/addrelease.py?json=true&committee=xxx&version=xxx&date=xxx