For Parsoid calls, treat preprocessing as starting in SOL state

* If Parsoid calls the preprocessor, initialize lineStart to true.
  Track this through:
  - parser function calls that return expandable template messages
    int: parser function is an example in core
    Extensions seem to define a number of other such parser functions
  - template-arg substitions
    So {{templatename|mytemplate}} with text {{{{{1}}}}} which
    is effectively a call to {{mytemplate}} continues to set sol-state
    to true across the expansion.

    See test "Preprocessor precedence 5: tplarg takes precedence over template"
    in preprocessor.txt which exercises this use case.

  - However, note that this is a best-faith effort because this flag is
    set while building the preprocessor DOM tree before templates are
    expanded. So, this is mostly a source syntax flag and constructs
    that expand to empty strings can blind the preprocessor to the true
    value of SOL state in the expanded string. This is true for both
    the legacy parser and Parsoid, and as such T2529 behavior is a hack
    with a set of associated edge cases.

* Parsoid models templates as independent documents as always starting
  in start-of-line state (and does some patch up for b/c reasons where
  this assumption fails). So, there is no reason to add newlines for
  some set of wikitext characters (per T2529) when Parsoid is involved.

* This lets us eliminate some hacks in Parsoid to strip these added newlines
  when Parsoid was already in SOL state but which then introduces edge cases.
  See discussion in T382464 where Parsoid currently cannot distinguish
  between a couple of test cases.

* But, with this change, where Parsoid no longer gets a newline added,
  Parsoid doesn't have to heuristically remove the newline (and
  incorrectly as in the edge case in the bug report) which eliminates
  the edge case from the bug report.

* This change has to be backed by a change in Parsoid to undo the T2529
  newline removal hack in TokenStreamPatcher to ensure Parsoid CI
  doesn't break with this change.

* To let us safely test this in Parsoid's round-trip testing and safely
  (and conservatively) roll this out to production, this change is
  backed by a new config flag (ParsoidTemplateExpansionMode) which
  defaults to false.

  We unconditionally set this to true in the ParserTestRunner for all
  parser tests.

  This flag will be removed once we roll out this change and the
  Parsoid change to production.

Bug: T382464
Change-Id: I194a9550500bf7ece215791c51d6feb78a80b1a8

1 files changed, 2 insertions, 0 deletions

diff --git a/includes/config-schema.php b/includes/config-schema.php
index 27a36da6dc92..d17047289d03 100644
--- a/includes/config-schema.php
+++ b/includes/config-schema.php
@@ -793,6 +793,7 @@ return [
 				'useSelser' => true,
 			],
 			'ParsoidFragmentSupport' => false,
+			'ParsoidNewTemplateExpansionMode' => false,
 			'ParserEnableLegacyMediaDOM' => false,
 			'ParserEnableLegacyHeadingDOM' => false,
 			'UseContentMediaStyles' => false,
@@ -2808,6 +2809,7 @@ return [
 				'boolean',
 				'string',
 			],
+			'ParsoidNewTemplateExpansionMode' => 'boolean',
 			'NoFollowNsExceptions' => 'array',
 			'NoFollowDomainExceptions' => 'array',
 			'EnableMagicLinks' => 'object',