Revcheck library and QA tools for translation synchronization

CODEOWNERS

@@ -0,0 +1,11 @@
+# The following volunteers have self-identified as subject matter experts
+# or interested parties over a particular area of this repository.
+# While requesting a review from someone does not obligate that person to
+# review a pull request, these reviewers might have valuable knowledge of
+# the problem area and could aid in deciding whether a pull request is ready
+# for merging.
+#
+# For more information, see the GitHub CODEOWNERS documentation:
+# https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners
+
+/scripts/translation/ @alfsb

README.md

@@ -143,3 +143,8 @@ and find issues with it, they are located in the `scripts/qa/` directory.
 There might be some more just in `scripts/` but they need to be checked if they
 are still relevant and/or given some love.
 
+# Translation Tools
+
+There are also various scripts to ensure the quality and synchrony of
+documentation translations, located in the `scripts/translation/` directory.
+

scripts/translation/.gitignore

@@ -0,0 +1,2 @@
+# Persistent data shared between scripts
+.cache/

scripts/translation/README.md

@@ -0,0 +1,111 @@
+# Some useful scripts for maintaining translation consistency of manual
+
+Some of these scripts only test some file contents or XML structure
+of translated files against their equivalents on `en/` directory.
+Others will try modify the translations in place, changing the
+translated files. Use with care.
+
+Not all translations are identical, or use the same conventions.
+So not all scripts will be of use for all translations. The
+assumptions of each script are described in each file.
+
+The `lib/` directory contains common code and functionality
+across these scripts.
+
+Before using the scripts, it need be configured:
+```
+php doc-base/scripts/translation/configure.php $LANG_DIR
+```
+
+## qarvt.php
+
+`qarvt.a.php` checks if all translated files have revtags in the
+expected format.
+
+## qaxml.a.php
+
+`qaxml.a.php` checks if all updated translated files have
+the same tag-attribute-value triples. Tag's attributes are extensively
+utilized in manual for linking and XIncluding. Translated files with
+missing os mistyped attributes may cause build failing or missing
+parts not copied by XIncludes.
+
+## qaxml.e.php
+
+`qaxml.e.php` checks if all updated translated files have
+the same external entities as the original files. Unbalanced entities
+may indicate mistyped or wrongly traduced parts.
+
+## qaxml.p.php
+
+`qaxml.p.php` checks if all updated translated files have
+the same processing instructions as the original files. Unbalanced entities
+may cause compilation errors, as they are utilized on manual in the build
+process.
+
+## qaxml.t.php
+
+`qaxml.t.php` checks if all updated translated files have
+the same tags as the original files. Different number of tags between
+source texts and target translations may cause compilation errors.
+
+Usage: `php qaxml.t.php [--detail] [tag[,tag]]`
+
+`[tag[,tag]]` is a comma separated tag list to check their
+contents, as some tag's contents are expected *not* be translated.
+
+`--detail` will also print line defintions of each mismatched tag,
+to facilitate bitsecting.
+
+## Suggested execution
+
+Structural checks:
+
+```
+php doc-base/scripts/translation/configure.php $LANG_DIR
+
+php doc-base/scripts/translation/qarvt.php
+
+php doc-base/scripts/translation/qaxml.a.php
+php doc-base/scripts/translation/qaxml.e.php
+php doc-base/scripts/translation/qaxml.p.php
+php doc-base/scripts/translation/qaxml.t.php
+```
+Tags where is expected no translations:
+```
+php doc-base/scripts/translation/qaxml.t.php acronym
+php doc-base/scripts/translation/qaxml.t.php classname
+php doc-base/scripts/translation/qaxml.t.php constant
+php doc-base/scripts/translation/qaxml.t.php envar
+php doc-base/scripts/translation/qaxml.t.php function
+php doc-base/scripts/translation/qaxml.t.php interfacename
+php doc-base/scripts/translation/qaxml.t.php parameter
+php doc-base/scripts/translation/qaxml.t.php type
+php doc-base/scripts/translation/qaxml.t.php classsynopsis
+php doc-base/scripts/translation/qaxml.t.php constructorsynopsis
+php doc-base/scripts/translation/qaxml.t.php destructorsynopsis
+php doc-base/scripts/translation/qaxml.t.php fieldsynopsis
+php doc-base/scripts/translation/qaxml.t.php funcsynopsis
+php doc-base/scripts/translation/qaxml.t.php methodsynopsis
+```
+Tags where is expected few translations:
+```
+php doc-base/scripts/translation/qaxml.t.php code
+php doc-base/scripts/translation/qaxml.t.php computeroutput
+php doc-base/scripts/translation/qaxml.t.php filename
+php doc-base/scripts/translation/qaxml.t.php literal
+php doc-base/scripts/translation/qaxml.t.php varname
+```
+
+# Migration
+
+## Maintainers with spaces
+
+The regex on `RevtagParser` was narrowed to not accept maintainer's names
+with spaces. This need to be confirmed on all active translations, or
+the regex modified to accept spaces again.
+
+## en/chmonly
+
+`en/chmonly` is ignored on revcheck, but it appears translatable. If it's a
+`en/` only directory, this should be uncommented on RevcheckIgnore.

scripts/translation/configure.php

@@ -0,0 +1,29 @@
+<?php
+/**
+ *  +----------------------------------------------------------------------+
+ *  | Copyright (c) 1997-2023 The PHP Group                                |
+ *  +----------------------------------------------------------------------+
+ *  | This source file is subject to version 3.01 of the PHP license,      |
+ *  | that is bundled with this package in the file LICENSE, and is        |
+ *  | available through the world-wide-web at the following url:           |
+ *  | https://www.php.net/license/3_01.txt.                                |
+ *  | If you did not receive a copy of the PHP license and are unable to   |
+ *  | obtain it through the world-wide-web, please send a note to          |
+ *  | license@php.net, so we can mail you a copy immediately.              |
+ *  +----------------------------------------------------------------------+
+ *  | Authors:     André L F S Bacci <ae php.net>                          |
+ *  +----------------------------------------------------------------------+
+ *  | Description: Generate cached data for revcheck and QA tools.         |
+ *  +----------------------------------------------------------------------+
+ */
+
+require_once __DIR__ . '/lib/all.php';
+
+if ( count( $argv ) < 2 || in_array( '--help' , $argv ) || in_array( '-h' , $argv ) )
+{
+    fwrite( STDERR , "Usage: {$argv[0]} [lang_dir]\n\n" );
+    fwrite( STDERR , "See https://github.com/php/doc-base/tree/master/scripts/translation#readme for more info.\n" );
+    return;
+}
+
+new RevcheckRun( 'en' , $argv[1] , true );

scripts/translation/lib/CacheFile.php

@@ -0,0 +1,57 @@
+<?php
+/**
+ *  +----------------------------------------------------------------------+
+ *  | Copyright (c) 1997-2023 The PHP Group                                |
+ *  +----------------------------------------------------------------------+
+ *  | This source file is subject to version 3.01 of the PHP license,      |
+ *  | that is bundled with this package in the file LICENSE, and is        |
+ *  | available through the world-wide-web at the following url:           |
+ *  | https://www.php.net/license/3_01.txt.                                |
+ *  | If you did not receive a copy of the PHP license and are unable to   |
+ *  | obtain it through the world-wide-web, please send a note to          |
+ *  | license@php.net, so we can mail you a copy immediately.              |
+ *  +----------------------------------------------------------------------+
+ *  | Authors:     André L F S Bacci <ae php.net>                          |
+ *  +----------------------------------------------------------------------+
+ *  | Description: Class to handle data persistence.                       |
+ *  +----------------------------------------------------------------------+
+ */
+
+require_once __DIR__ . '/all.php';
+
+class CacheFile
+{
+    const CACHE_DIR = __DIR__ . '/../.cache';
+
+    private string $filename;
+
+    function __construct( string $file )
+    {
+        $this->filename = CacheFile::prepareFilename( $file , true );
+    }
+
+    public function load( mixed $init = null )
+    {
+        if ( file_exists( $this->filename ) == false )
+            return $init;
+        $data = file_get_contents( $this->filename );
+        return unserialize( gzdecode( $data ) );
+    }
+
+    public function save( $data )
+    {
+        $contents = gzencode( serialize( $data ) );
+        file_put_contents( $this->filename , $contents );
+    }
+
+    public static function prepareFilename( string $file , bool $createCacheDirs = false )
+    {
+        if ( str_starts_with( $file , '/' ) )
+            return $file;
+        $outPath = CacheUtil::CACHE_DIR . '/' . dirname( $file );
+        $outFile = rtrim( $outPath , '/' ) . '/' . $file;
+        if ( $createCacheDirs && file_exists( $outPath ) == false )
+            mkdir( $outPath , 0777 , true );
+        return $outFile;
+    }
+}

scripts/translation/lib/CacheUtil.php

@@ -0,0 +1,51 @@
+<?php
+/**
+ *  +----------------------------------------------------------------------+
+ *  | Copyright (c) 1997-2023 The PHP Group                                |
+ *  +----------------------------------------------------------------------+
+ *  | This source file is subject to version 3.01 of the PHP license,      |
+ *  | that is bundled with this package in the file LICENSE, and is        |
+ *  | available through the world-wide-web at the following url:           |
+ *  | https://www.php.net/license/3_01.txt.                                |
+ *  | If you did not receive a copy of the PHP license and are unable to   |
+ *  | obtain it through the world-wide-web, please send a note to          |
+ *  | license@php.net, so we can mail you a copy immediately.              |
+ *  +----------------------------------------------------------------------+
+ *  | Authors:     André L F S Bacci <ae php.net>                          |
+ *  +----------------------------------------------------------------------+
+ *  | Description: Common functions do load and save to cache files.       |
+ *  +----------------------------------------------------------------------+
+ */
+
+require_once __DIR__ . '/all.php';
+
+class CacheUtil
+{
+    const CACHE_DIR = __DIR__ . '/../.cache';
+
+    public static function load( string $path , string $file )
+    {
+        $filename = CacheUtil::prepareFilename( $path , $file , true );
+        if ( file_exists( $filename ) == false )
+            return null;
+        $data = file_get_contents( $filename );
+        return unserialize( $data );
+    }
+
+    public static function save( string $path , string $file , $data )
+    {
+        $outFile = CacheUtil::prepareFilename( $path , $file , true );
+        $contents = serialize( $data );
+        file_put_contents( $outFile , $contents );
+    }
+
+    public static function prepareFilename( string $path , string $file , bool $createDirs = false )
+    {
+        $baseDir = CacheUtil::CACHE_DIR;
+        $outPath = rtrim( $baseDir , '/' ) . '/' . $path;
+        $outFile = rtrim( $outPath , '/' ) . '/' . $file;
+        if ( $createDirs && file_exists( $outPath ) == false )
+            mkdir( $outPath , 0777 , true );
+        return $outFile;
+    }
+}

scripts/translation/lib/GitDiffParser.php

@@ -0,0 +1,26 @@
+<?php
+/**
+ *  +----------------------------------------------------------------------+
+ *  | Copyright (c) 1997-2023 The PHP Group                                |
+ *  +----------------------------------------------------------------------+
+ *  | This source file is subject to version 3.01 of the PHP license,      |
+ *  | that is bundled with this package in the file LICENSE, and is        |
+ *  | available through the world-wide-web at the following url:           |
+ *  | https://www.php.net/license/3_01.txt.                                |
+ *  | If you did not receive a copy of the PHP license and are unable to   |
+ *  | obtain it through the world-wide-web, please send a note to          |
+ *  | license@php.net, so we can mail you a copy immediately.              |
+ *  +----------------------------------------------------------------------+
+ *  | Authors:     André L F S Bacci <ae php.net>                          |
+ *  +----------------------------------------------------------------------+
+ *  | Description: Parse `git diff` to complement file state.              |
+ *  +----------------------------------------------------------------------+
+ */
+
+require_once __DIR__ . '/all.php';
+
+class GitDiffParser
+{
+    public static function parseNumstatInto( string $dir , RevcheckFileInfo $file )
+    {}
+}

scripts/translation/lib/GitLogParser.php

@@ -0,0 +1,93 @@
+<?php
+/**
+ *  +----------------------------------------------------------------------+
+ *  | Copyright (c) 1997-2023 The PHP Group                                |
+ *  +----------------------------------------------------------------------+
+ *  | This source file is subject to version 3.01 of the PHP license,      |
+ *  | that is bundled with this package in the file LICENSE, and is        |
+ *  | available through the world-wide-web at the following url:           |
+ *  | https://www.php.net/license/3_01.txt.                                |
+ *  | If you did not receive a copy of the PHP license and are unable to   |
+ *  | obtain it through the world-wide-web, please send a note to          |
+ *  | license@php.net, so we can mail you a copy immediately.              |
+ *  +----------------------------------------------------------------------+
+ *  | Authors:     André L F S Bacci <ae php.net>                          |
+ *  +----------------------------------------------------------------------+
+ *  | Description: Parse `git log` to complement file state.               |
+ *  +----------------------------------------------------------------------+
+ */
+
+require_once __DIR__ . '/all.php';
+
+class GitLogParser
+{
+    static function parseInto( string $lang , RevcheckFileList & $list )
+    {
+        $cwd = getcwd();
+        chdir( $lang );
+        $fp = popen( "git log --name-only" , "r" );
+        $hash = "";
+        $date = "";
+        $skip = false;
+        while ( ( $line = fgets( $fp ) ) !== false )
+        {
+            // new commit block
+            if ( substr( $line , 0 , 7 ) == "commit " )
+            {
+                $hash = trim( substr( $line , 7 ) );
+                $date = "";
+                $skip = false;
+                continue;
+            }
+            // datetime of commit
+            if ( strpos( $line , 'Date:' ) === 0 )
+            {
+                $line = trim( substr( $line , 5 ) );
+                $date = strtotime( $line );
+                continue;
+            }
+            // other headers
+            if ( strpos( $line , ': ' ) > 0 )
+                continue;
+            // empty lines
+            if ( trim( $line ) == "" )
+                continue;
+            // commit message
+            if ( str_starts_with( $line , '    ' ) )
+            {
+                // commits with this mark are ignored
+                if ( stristr( $line, '[skip-revcheck]' ) !== false )
+                    $skip = true;
+                continue;
+            }
+            // otherwise, a filename
+            $filename = trim( $line );
+            $info = $list->get( $filename );
+
+            // untracked file (deleted, renamed)
+            if ( $info == null )
+                continue;
+
+            // the head commit
+            if ( $info->head == "" )
+            {
+                $info->head = $hash;
+                $info->date = $date;
+            }
+
+            // after, only tracks non skipped commits
+            if ( $skip )
+                continue;
+
+            // the diff commit
+            if ( $info->diff == "" )
+            {
+                $info->diff = $hash;
+                $info->date = $date;
+            }
+        }
+
+        pclose( $fp );
+        chdir( $cwd );
+    }
+}