jtfm-notes.txt

* description

  create and run certain "tests" that live in comment forms.

* rationale

  * want some meaningful "tests" / "usages" to co-evolve with
    development from as early as desired with little activation
    effort.

  * easily record some of one's intent / actually tested expressions
    and results for later testing and reference.

* non-goals

  * not meant as a single-stop testing approach.  meant to be usable
    alongside other testing tools, frameworks, etc.

* random

  * port the batch-test.janet script that lives under
    ~/src/janet-usages-as-tests...or at least give something similar a
    home in jtfm's project directory?  make running it part of the pre
    commit steps.  better to have "the new version" try to use the
    latest jtfm instead of whatever is "bundled" in each repository --
    otherwise it's not much of a test of the latest jtfm.

  * any value in trying to use fibers / threads to help with
    processing different types of output?

* limits

  * unrecoverable errors provide feedback to the user that is still on
    the rough side.

  * not able to turn off "administrative output", e.g. path being
    tested, "task complete" types of output, etc.  this works against
    features like having "update" lead to updated content being sent
    to stdout because it gets "entangled" among other output.

    * if admin output went to stderr instead that might help

    * if update output went to stderr that might help

    * for testing purposes, might be better if all output is
      structured as "data" (if possible).  easy to parse and make
      sense of?  how about "logging" of events?  seems ok for "small"
      data but would this be practical for "larger" data?

      may be not all output?  which output would it make sense to make
      "structured" then?  the runner essentially gives structured
      output already.  can the output from each runner instance be
      collected into one big piece?  is that desirable?

    * consider classifying output by purpose instead of by
      destination.  e.g. "status" or "log" might indicate reporting to
      the user what is happening.  "error" might indicate a problem.
      "warning" might indicate a potential problem.  "data" or
      "output" might indicate intended output of program (not "meta").

      however, in the case of "error", the flow of the program
      probably needs to change so its handling may need to differ.

      by classifying this way, the decisions about where to send each
      type can be modified programmatically without changing code,
      i.e. perhaps options can be used to configure.  one extreme
      might be for each type of output to have its own file.  another
      extreme is to send all output to one place.  in between are the
      options to send some types to one place, some other types
      elsewhere, and yet other types to still yet another location.

    * is it practical and desirable to rework the code so that all of
      the output code is "at the edges"?

  * no programmatic testing for cli invocation.  one thing that seems
    to be a barrier is the uncertainty involved in testing against the
    filesystem.

    default to using tmp subdirectory of repository.  possibly allow
    path specification?

    * test :update-first capability

    * test :update capability

    * test feedback from use of unreadable value

    * test passing all tests capability

    look at some of jackal's usage testing for cli-ish example.

  * source files are not linted before creating test files.  linting
    might issue warnings for legitimate files so better to make this
    optional?

  * test data cannot be output in "raw" jdn form

  * only test failure data (plus total number of tests) is
    communicated back to the runner.

  * no distinction between error result strings and ordinary strings
    in test output is made.  the status is now available so making a
    distinction may be possible.

  * how 3rd party abstract types print is decided by the implementor
    of that abstract type and does not need to follow any particular
    convention(?).  for example, a "set" abstract type could print in
    a way such that it looks like a tuple.  this might be problematic,
    because the printed representation cannot be relied upon as an
    indicator of the "type" of value.  however, perhaps the `type`
    function could be used to make certain distinctions.

  * printing of tables with prototypes does not show prototype info.
    consequently, checking such table values requires extra work.  can
    use table/getproto and/or struct/getproto to check for existence
    of prototype table/struct.

  * updating of expected values is still rough

    * not able to target particular test for updating, say via an
      explicit line number.  does first one or all.

    * feedback from updating doesn't look very nice

    * one issue that seems to hold back this feature is the lack of an
      appropriate pretty printer.  note that this only requires pretty
      printing of data, and not "code", so it's not exactly the same
      problem that a code formatter tries to address and perhaps it's
      somewhat easier?

    * non-pretty-printed values are used.  this might lead to manually
      reformatting a fair number of actual values...which might be
      opportunity to observe the various processes of reformatting.
      also, may be it wouldn't be that common to have to reformat too
      many results?

  * there is no checking of unsupported / potentially problematic
    constructs (e.g. forms that look like but are not tests, disabled
    tests, etc.).  might be nice to have a checker / linter one can
    run to get reports.

  * no stacktrace when error for actual value

    * not a clear win to have a stacktrace because the
      actual code being run is the transformed code and
      this might be confusing (could the output be
      tweaked appropriately?)

    * current recommendation is to reproduce the error in one's
      editor.  the line number (and file) of the relevant test
      indicator should be in the test run output.  doing this should
      yield appropriate stack traces.

  * errors related to test file creation and running may be hard to
    understand.  try to watch out for and document specific cases.

  * test failure leaves the created test file in place.  it does not
    seem to be removed automatically on next execution.  there is an
    option to overwrite test files but it is not the default.

* features

  * "tests" can be written without installing any tool or library,
    e.g.:

      (comment

        (my-func arg1)
        # =>
        :my-nice-result

        )

    the "actual" portion of the test (the bit above the `# =>`) can be
    manually "executed" via typical editor tooling for janet
    (e.g. "send expression to repl").

  * more convenient running of tests can be had via an additional
    file, `jtfm.janet`.  e.g. "tests" within a file can be executed
    by:

      ./jtfm.janet src/my-file.janet

  * single file deployment.  `jtfm.janet` does not need to be
    "installed", nor does it add a dependency to one's project.  it
    just needs to be on `PATH` or it can sit within or under one's
    project directory (recommended).

  * for use with jpm test, jeep test, etc.  two additional files need
    to be created.

    one small file under a project's `test` directory to trigger the
    runner, e.g. if `jtfm.janet` is in one's project root:

      (import ../jtfm)

      (jtfm/main)

    and one in the project root named `.jtfm.jdn` to configure what to
    test:

      {
       # describes what to test - file and dir paths
       :includes ["src" "obj" "jtfm.janet"]
       # describes what to skip - file paths only
       :excludes []
      }

    with these bits in place, `jpm test`, `jeep test`, etc. should
    work.

  * test expected values can be updated.  either first found or all.

  * `(def ...)` and similar "forms that involve definitions" are
    usable a "test expressions":

      (def a 1)
      # =>
      1

    this was a limitation in previous incarnations of the basic idea
    (e.g. judge-gen, usages-as-tests, janet-usages-as-tests,
    janet-ex-as-tests).

* questions

  * what benefits were there be to using a separate channel of
    communication that:

    * is multiplatform
    * does not involve creating and cleaning up temp files
    * not overly complex

    to communicate either test output or test result reporting?

  * is it a flaw that file paths within usages seem to be relative to
    the root of the project even though relative paths for imports are
    relative to the source file itself?  it does seem confusing...
    however, it seems to be fundamental because the current working
    directory of the janet process is the root of the project
    directory.  this seems inescapable...not sure though.

  * is it possible to avoid the problem of `jpm test` executing all
    .janet files within the test/ subdir?  currently, placing
    test-related files in test/ is awkward because of `jpm test`'s
    behavior.
  
    * change file extension to be something other than .janet so that
      `jpm test` and friends don't trigger the code

    * can the module loader be tweaked to load the files with a
      different extension?

* hidden features

  * {:overwrite true} overwrites old test files (typically those that
    has failures)

  * {:includes ...} and {:excludes ...} can also be used to augment
    paths specified on the command line or via the configuration file.

  * {:update-first true} attempts updating of first test's actual
    value

  * {:update true} attempts updating of all tests' actual values

  * NO_COLOR environment variable support

  * `( comment ...)` (note the space between the opening paren and the
    letter "c") disables testing

  * transforming `# =>` into `# = >` disables the use of the
    corresponding code as a test

* semi-recurring tasks

  * just reading the existing code and this file on a tablet for
    review purposes

  * look for code that could do with more testing.

* pre commit steps

  1. run jell and examine results

     * if problem detected, investigate, take action, and go to step 1

  2. run tests and examine results

     * if problem detected, investigate, take action, and go to step 1

  3. batch test across all projects that use jtfm

     * if problem detected, investigate, take action, and go to step 1

  4. assemble and review staging for commit

     * if it looks like changes are needed, make them and go to step 1

     * if not, commit and push