1.0 releaseHEADmain

2 files changed, 171 insertions, 58 deletions

diff --git a/doc/dev/index.rst b/doc/dev/index.rst
index 523e6a9..de8b996 100644
--- a/doc/dev/index.rst
+++ b/doc/dev/index.rst
@@ -1,3 +1,13 @@
+.. raw:: html
+
+   <style>
+   td {
+     white-space: normal !important;
+     vertical-align: unset !important;
+   }
+   </style>
+
+
 ##############
 Developer docs
 ##############
@@ -46,6 +56,9 @@ Nomenclature
    patchset
      A set of files that describe how to change the target source tree(s), placed in the same structure as the original source tree.
 
+   patchspec
+     A YAML specification that configures how patchtree should handle (other) inputs.
+
    patch
      A single file (usually ``.patch`` or ``.diff``) that lists changes to make to one specific target directory.
 
@@ -56,7 +69,7 @@ Building clean patches
 In order to generate clean patches files, patchtree needs
 
 * the (original) :term:`target` source tree (either as a folder or .zip file)
-* a set of :term:`inputs`
+* a set of :term:`inputs` (i.e. the patchset)
 
 The basic syntax of the patchtree CLI is as follows:
 
@@ -67,36 +80,114 @@ The basic syntax of the patchtree CLI is as follows:
 By default, the resulting patch is written to the standard output.
 This behavior, along with many other default behaviors can be changed through the command-line arguments (see ``--help``) or the `configuration file <ptconfig_>`_.
 
-************************
-Writing patchset sources
-************************
+.. _patchspec:
+
+***********************
+Writing patchset inputs
+***********************
 
-Each patchset source file is compared to the target source file of the same name, and the resulting diff is output in the clean patch.
+Each patchset input file is compared using difflib's unified_diff algorithm to the target source file of the same name, and the resulting diff is output in the clean patch.
 This means that the default behavior of files placed in the patchset is to add or replace any file in the target source tree.
+Because most of the time only small adjustments have to be made to the target sources, patchtree implements mechanisms for semantically describing what changes should be applied.
+This is achieved by writing a :term:`patchspec`, which is located either at the start of a patchset input file or as a standalone .yml file in the patchset tree.
 
-Because most of the time only small adjustments have to be made to the target sources, patchtree uses so-called processors.
-Every patchset source is first processed by 0 or more processors, which transform the input's content before it is compared to the target file's content.
-This mechanism allows you to describe changes *semantically* so they can apply to multiple versions of-- or variations in the target.
+.. code-block:: yaml
+   :caption: Patchspec header
+
+   --- !patchspec
+   foo: bar
+   ...
+
+   remaining file content
+
+.. note::
+
+   While patchspec headers look similar to `front matter <https://frontmatter.codes>`_, they are actually regular YAML documents (and must therefore be delimited from the remaining file content using ``...`` instead of ``---``).
+   Any valid YAML document at the beginning of a file which explicitly defines a patchspec (using the ``!patchspec`` tag) is processed.
+   If a YAML file is present in the patchset but does not define a patchspec, it will be treated as any other text file!
+
+.. code-block:: yaml
+   :caption: Standalone patchspec
+
+   !patchspec
+   foo: bar
+
+The output from standalone patchspecs are compared against the file in the target of the same name **after the .yml/.yaml extension is removed**.
+The file extensions trimmed from standalone patchspecs can be configured using the `configuration file <ptconfig_>`_'s :any:`patchspec_extensions <Config.patchspec_extensions>` value.
+Note that a file must both have an extension from the aforementioned list as well as contain a valid patchspec in order to be handled as non-literal input.
+
+=========
+YAML tags
+=========
+
+In addition to the default tags supported by `PyYAML <https://pyyaml.org/>`_, patchtree supports the following tags:
 
+.. list-table::
+   :header-rows: 1
+
+   * - Name
+     - YAML type
+     - Description
+   * - ``!patchspec``
+     - mapping
+     - Provides patchspec type and value validation and is required in order to make patchtree use the patchspec.
+   * - ``!target``
+     - scalar
+     - Input specification for a file in the target.
+       If no filename is passed to this tag, the current filename is used.
+       Creates a :any:`TargetFileInputSpec`.
+   * - ``!input``
+     - scalar
+     - Input specification for a file in the patchset (relative to current file).
+       If no filename is passed to this tag, the current filename is used.
+       Creates a :any:`PatchsetFileInputSpec`.
+
+       .. note::
+
+          Files specified using ``!input`` are no longer treated as a literal (unprocessed) input and will not be compared to a target file of the same name.
+
+
+================
+Patchspec format
+================
+
+The patchspec format currently takes the following keys:
+
+.. code-block:: yaml
+
+   !patchspec
+   # List of processors to apply (in order defined)
+   processors: list[Processor]()
+
+
+.. _patchspec_tags:
+
+==========
 Processors
 ==========
 
-Processors are indicated using the hash symbol (``#``) in the filename, and each process the input's contents in a chain.
-Processors may optionally take argument(s) separated by a comma (``,``), and arguments may optionally take a value delimited by an equal sign (``=``)
-After processing, the resulting file content is compared to the target source's content using difflib's unified_diff algorithm.
+Every patchset source is first processed by 0 or more processors, which transform the input's content before it is compared to the target file's content.
+This mechanism allows you to describe changes *semantically* so they can apply to multiple versions of-- or variations in the target.
+
+Each processor takes an input and output similar to the standard input/output of command-line processes.
+The only difference with patchtree is that the input/output is file-based, and carries metadata about the file's mode and whether it is binary or not.
+The processors can be thought of as piped commands in the sense that (by default) each processor's input is either (a) the content of the patchset file for the first processor, or (b) the output received from the previous processor.
+Some processors also take secondary input(s), usually under the name *target*.
 
-For example:
+For each process, the input and/or target can be manually specified using an *input spec*.
+The input spec's default value is to use the output of the previous processor.
+It can also be set to a constant string (inside the YAML patchspec) or an existing file inside the patchset or target using YAML tags (see :ref:`patchspec_tags`).
 
-.. code:: none
+As an example, the following patchspec will set the mode of the file it is named as (with an added .yml extension) to 755 (executable):
 
-   sdk/middleware/adapters/src/ad_crypto.c#cocci#jinja
-   \_____________________________________/\__________/
-           target source file path         processors
+.. code-block:: yaml
 
-In the above example, the input is first processed by :ref:`jinja <process_jinja>`, and the resulting file content is piped into :ref:`Coccinelle <process_cocci>` as if a file with the output from jinja existed under the name ``ad_crypto.c#cocci``.
-Coccinelle will in this case output a modified copy of ``ad_crypto.c``, which will be compared to the original to produce the diff for this file.
+   !patchspec
+   processors:
+     - id: touch
+       mode: 0755
 
-The processors included with patchtree are documented on the :ref:`processors` page.
+The processors included with patchtree, including any configuration options they take, are documented on the :ref:`processors` page.
 Custom processors can be created by inheriting from the base :any:`Process` class and registering through the `configuration file <ptconfig_>`_'s :any:`processors <Config.processors>` value.
 
 .. _ptconfig:
diff --git a/doc/dev/processor.rst b/doc/dev/processor.rst
index 018d216..640f44c 100644
--- a/doc/dev/processor.rst
+++ b/doc/dev/processor.rst
@@ -5,25 +5,28 @@ Processors
 ##########
 
 This page lists all built-in processor types along with descriptions of what they do and which options they take.
-On this page, **output** refers to what the processor returns, while **input** refers to how the processor treats its input.
-This input is either (a) the content of the patchset file for the first processor, or (b) the output received from the previous processor.
-**Arguments** are any options explicitly given to the processor through the filename, e.g. ``filename#processor,arg,arg2=value,arg3#processor2``.
-Note that some processors may take positional arguments, while others may use key/value based options instead.
 
-.. _process_id:
+.. _process_touch:
 
-********
-Identity
-********
+*****
+Touch
+*****
 
-The identity processor is used to "touch" files, change the mode of existing files, or add arbitrary identifiers to patchset source filenames by passing arbitrary arguments.
+The touch processor is used to create files (similar to ``touch``) or change the mode of existing files (similar to ``chmod``).
 
-:Class: :any:`ProcessIdentity`
-:Identifier: ``id``
-:Input: Ignored.
+:Class: :any:`TouchProcess`
+:Input: Content of new file.
 :Output:
-  A file with the *content* of the target file and *mode* of the patchset input.
-:Arguments: Any arguments passed to this processor are ignored.
+  The input file as-is, with the following exceptions:
+
+  - If the input is empty, it will be set to an empty (but existent) file.
+  - If the ``mode`` option is set, it will be used instead of the input mode.
+:Options:
+  .. code-block:: yaml
+
+     - id: "touch"
+       input: ProcessInputSpec(required=False)
+       mode: int(required=False, default=<input mode>)
 
 .. _process_cocci:
 
@@ -37,11 +40,17 @@ The Coccinelle processor uses Coccinelle to apply patch(es) in the SmPL (Semanti
 
    In order to use this processor, Coccinelle must be installed and ``spatch`` must be available in ``$PATH``.
 
-:Class: :any:`ProcessCoccinelle`
-:Identifier: ``cocci``
+:Class: :any:`CoccinelleProcess`
 :Input: Coccinelle's SmPL input.
-:Output: The contents of the target file after being processed by Coccinelle (not the diff returned by Coccinelle).
-:Arguments: Reserved.
+:Target: File content to apply patch to (current patchspec's target file by default).
+:Output: The content of the target file after being processed by Coccinelle (not the diff returned by Coccinelle).
+:Options:
+  .. code-block:: yaml
+
+     - id: "cocci"
+       input: ProcessInputSpec(required=False)
+       target: ProcessInputSpec(required=False)
+
 
 .. _process_jinja:
 
@@ -51,16 +60,19 @@ Jinja template
 
 The Jinja processor passes the input through the Jinja2 templating engine.
 
-:Class: :any:`ProcessJinja2`
-:Identifier: ``jinja``
+:Class: :any:`Jinja2Process`
 :Input: Jinja template code.
 :Output: The input after being processed by Jinja.
-:Arguments: Reserved.
+:Options:
+  .. code-block:: yaml
+
+     - id: "jinja"
+       input: ProcessInputSpec(required=False)
 
 .. note::
 
-   Template variables are generated through the :any:`get_template_vars <ProcessJinja2.get_template_vars>` method.
-   This method returns an empty dict by default, and is meant to be implemented by implementing a custom class that derives from ProcessJinja2 and registering it through the :ref:`configuration file <ptconfig>`.
+   Template variables are generated through the :any:`get_template_vars <Jinja2Process.get_template_vars>` method.
+   This method returns an empty dict by default, and is meant to be implemented by implementing a custom class that derives from :any:`Jinja2Process` and registering it through the :ref:`configuration file <ptconfig>`.
 
 .. _process_exe:
 
@@ -68,19 +80,21 @@ The Jinja processor passes the input through the Jinja2 templating engine.
 Executable
 **********
 
-The executable processor runs the input as an executable, passes the target file to its standard input, and returns its standard output.
+The executable processor passes its input to an executable and returns its standard output.
 
-:Class: :any:`ProcessExec`
-:Identifier: ``exec``
-:Input:
-  Executable script.
+:Class: :any:`ExecProcess`
+:Input: Input passed to the standard input of the command.
+:Output: Any content written to the standard output by the executable.
+:Options:
+  .. code-block:: yaml
 
-  .. important::
+     - id: "exec"
+       cmd: str() | list[str]()
+       input: ProcessInputSpec(required=False)
 
-     The executable *must* contain a shebang line to specify what interpreter to use.
+  .. note::
 
-:Output: Any content written to the standard output by the executable.
-:Arguments: Reserved.
+     If the ``cmd`` option is a string, it is split **using shell syntax rules**.
 
 .. _process_merge:
 
@@ -90,13 +104,21 @@ Merge
 
 The merge processor merges the input with the target file, such that changes are combined with the target instead of replacing the target.
 
-:Class: :any:`ProcessMerge`
-:Identifier: ``merge``
-:Input: Content to merge.
+:Class: :any:`MergeProcess`
+:Input: Content to merge (A).
+:Target: Content to merge (B).
 :Output: Merged changes.
-:Arguments: Positional.
+:Options:
+  .. code-block:: yaml
+
+     - id: "merge"
+       strategy: enum() # see below
+       input: ProcessInputSpec(required=False)
+       target: ProcessInputSpec(required=False)
 
-  1. Merge strategy:
+==========
+Strategies
+==========
 
-     ``ignore``
-       Appends all lines from the input to the output excluding any lines already present in the output.
+``ignore``
+  Appends all lines from *input* to *target*, excluding any lines already present in *target*.