Template:Imported module list/doc

This is a transclusion template that provides the output of the LadyGeek developed python script below. The script uses the MediaWiki API to generate the list of Modules. History and background can be found on here.

Source

The python script was developed in Linux.

The arvlimit in the MediaWiki documentation states that the upper limit is 500 revisions, but the API sandbox says 5000. I kept the 5000 sandbox limit to avoid going over 500 in the near future. A warning will be printed every time API data is fetched; use it as a loop counter. A different number of entries will be returned each time (it's not evenly divided).

#!/usr/bin/python3
"""
   get_allrevisions_finiki.py

        Based on https://www.mediawiki.org/wiki/API:Allrevisions#Sample_code
        and https://www.mediawiki.org/wiki/API:Continue

        finiki.org API sandbox URL:
        https://www.finiki.org/wiki/Special:ApiSandbox#action=query&format=json&prop=&list=allrevisions&formatversion=2&arvprop=ids%7Ctimestamp&arvlimit=5000&arvnamespace=828&arvdir=older

        MediaWiki's API sandbox has a 500 return limit and breaks the output into separate pages.
"""

import requests,  datetime
from operator import itemgetter
import urllib.parse

S = requests.Session()

URL = "https://www.finiki.org/w/api.php"
out_file = "wiki_table_finiki.txt"

params = {
        "action": "query",
        "format": "json",
        "prop": "",
        "list": "allrevisions",
        "formatversion": "2",
        "arvprop": "ids|timestamp",
        "arvlimit": "5000",       #API sandbox has 5000 limit, limit is 500 (warning if > 500, but it works)
        "arvnamespace": "828",
        "arvdir": "older"
}

headers = {'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36'}

def query(params,  URL,  headers):
    lastContinue = {}
    while True:
        # Clone original params
        req = params.copy()
        # Modify it with the values returned in the 'continue' section of the last result.
        req.update(lastContinue)
        # Call API
        result = requests.get(url=URL, params=req, headers=headers).json()
        if 'error' in result:
            raise print(result['error'])
        if 'warnings' in result:
            print(result['warnings']['allrevisions'])
        if 'query' in result:
            yield result['query']['allrevisions']
        if 'continue' not in result:
            break
        lastContinue = result['continue']

# build results array
query_results = []
for result in query(params, URL, headers):
     query_results.append(result)

 # unpack 1 level
all_revisions = []
for i in range(len(query_results)):
     for j in range(len(query_results[i])):
         all_revisions.append(query_results[i][j])

 # build data table
table_data = []             # init
for n in range(len(all_revisions)):
    # extract latest revision timestamp as date object
    ISO_date = datetime.date.fromisoformat(all_revisions[n]['revisions'][0]['timestamp'][0:10])
    # append title, date converted to string as Month day, year
    list_pair = (all_revisions[n]['title'], ISO_date.strftime("%b %d, %Y"))
    table_data.append(list_pair )

# Sort by title, date remains in descending order (query order preserved)
sorted_by_title= sorted (table_data, key=itemgetter(0))

# Remove duplicate entries
table_rows = []        # init
table_rows.append(sorted_by_title[0])
for i in range(1,  len(sorted_by_title)):
    if (table_rows[-1][0] != sorted_by_title[i][0]):   # append if different title
        table_rows.append(sorted_by_title[i])

#write wikitext table
with open(out_file,  'w') as f:
    f.write('{{table alignment}}\n')
    f.write('{{mw-datatable}}\n')
    f.write('{| class = "wikitable sortable mw-datatable col2center"\n')
    f.write('! Module name\n')
    f.write('! Last imported/updated\n')
    f.write('! class=unsortable | Wikipedia\n')
    f.write('! Notes\n')
    for n in range(len(table_rows)):
        f. write('|-\n')
        f.write('| [[{0}]] || {1} || [https://en.wikipedia.org/w/index.php?title=:{2}&action=info#Edit_history View edit history] || \n'.format(table_rows[n][0],  table_rows[n][1], urllib.parse.quote(table_rows[n][0])))
    f.write('|}\n')

Note

The syntax highlighting, line numbers and keywords does not currently work in finiki. The tags <syntaxhighlight>...</syntaxhighlight> to produce line numbers and source language formatting require python3 on the server. It is not currently available due to server hosting limitation.