Quick How-To for command-line Helpers

2022-05-17


Note: This is a "quick how-to". The instructions are subjectively reduced to a minimum, to keep focus and clarity.


Setup for the first time


[1] Unpack the archive:

unzip -o __ARCHIVE__.zip OR 7z x -y __ARCHIVE__.zip

(optional) Verify the checksums against the official checksum values.

Do not use checksums from unknown sources.

(optional) On Linux: chmod +x bin/* or chmod u+x bin/*.

[2] Enjoy.

The --help command line argument can be used.

Common information for all Helpers

  • Some apps accept wildcard-patterns, some accept wildcard-expressions.
  • Same app can be used in scripts designed to work on Linux and on Windows.

Short description for each helper app

  • cpmv ( Copy/Move )

    • like mkdir -p && cp or mkdir -p && mv
  • lmt ( Last Modified Time )

    • like find -iname | grep | xargs | ls -t | head -1, faster
    • recursive search for files
    • accepts wildcard-patterns
    • print last modified time
  • lsrec ( List Recursive )

    • like find -iname | grep | xargs | ls, faster
    • recursive search for files
    • accepts wildcard-patterns
    • print the list of files and summary
  • repls ( Replace Strings )

    • like cat | awk replace > same_file, faster
    • by default a string is replaced/removed (case insensitive match)
    • uses wildcard expressions instead of regular expressions
  • rmrec ( Remove Recursive )

    • like find -iname | grep | xargs | rm -r, faster with "force"
    • recursive remove files and directories
    • accepts wildcard-patterns
    • print the list of files and summary
    • by default it waits for user confirmation, unless "force" is used
    • non-aggressive, no chmod/chown with "force" argument
  • runcaplog ( Run and Capture output in Log file )

    • like app | tee
    • captures stdout & stderr on same or separate files
    • successful capture depends on the redirects made by the shell or by the apps
  • sortf ( Sort File )

    • like sort > same_file, faster
    • only ascii sort for now
  • weg ( Wildcard Expressions Grep )

    • like grep, faster
    • uses wildcard expressions instead of regular expressions
  • cats ( Cat and Squeeze blanks )

    • like cat -s
    • force new-line after last line
  • realpathrec ( Real Path Recursive )

    • like realpath, no sensible change in output
    • it tries to show results in Windows, like in Linux
  • touchrec ( Touch file Recursive )

    • like 'mkdir -p && touch`
  • truncfile ( Truncate File to specified length )

    • like truncate
  • waitkey ( Wait for a Key or until specified time elapses )

    • like bash read -n 1
    • wait for a key on stdin
    • return code is the (1st) key-code
    • optionally wait few seconds
    • successful operation depends on the terminal/console

Note: Instructions for this tool end here.


Additional notes on capturing build-output

  • Background:
    • It is not so easy as it should be with log messages from parallel builds.
    • Log messages may be mixed, because of the way OS & apps handle file flush.
    • Using performance optimizations, independent pipe channels ... these help up to a point.
    • The consumer should be faster than the producer.
    • The (pipe) buffers should not reach the full capacity.
  • Why bother?
    • There can be weird issues caused by the build environment tools and/or setup.
    • Previous good logs can be compared with latest logs.
    • Logs from different build environments can be compared to find a build issue.
    • By forwarding messages on the original channel, the next processes in the pipe, including the IDE, receive what is expected.
    • Using a pipe3-like function in a script did not work as expected in many situations. Some messages were mixed when the system load was high.
  • Good news:
    • Usually it is good enough to capture the output from a parallel build with a shell/script.
  • Bad news:
    • Especially in hard times, when the system load is high, "Usually" is not the same as "Always", or at least not good enough.


Why 'runcaplog' exists?

  • Want to:
    • Capture build logs from stderr and stdout channels on separate files.
    • Allow each message to pass on its own channel.
    • Minimize the likelihood of messages being mixed in the log.
  • Solution: "runcaplog" helper.


Usual capture-output approaches:

  • Combine stdout and stderr, then capture to a log file.
    • There is a chance that the messages will be mixed in the log.
    • The IDE or other processes may interpret some unwanted messages and may give unwanted results.
  • Capture only one of stdout and stderr to a log file.
    • Only that channel can be analyzed later.


Usual capture-output implementations:

  • If the IDE/build-apps you use accept stderr messages on stdout, then it could be ok with ( "$@" ) 2>&1 | tee log-file

  • Else maybe the IDE/build-apps accept stdout messages on stderr, then it could be ok with

    ( "$@" ) 1>&2 | tee log-file

  • Else capture without combining pipe channels. There are several possible variants, depending on the shell/applications used.

    `... use your-favorite-shell implementation here ...'

  • Else capture one log per build action (compile,link,etc).

    • Less mixed messages & slower build overall & lots of independent logs.
  • Else ( build in IDE and save the log when the issue appears ) use a tool which can handle independent stdout & stdin, and in the same time captures the output to one log file, like "runcaplog" or a similar dedicated app.


Additional notes regarding the license file
  • With a "Default License", there is usually a small delay for non-free helper applications.
    • Delays can add up quickly when the application is run for batch processing, several times per second/minute.
  • No delays during Happy Hours.
  • You can purchase an "Active License" to eliminate delays.

Previous Article Next Article

..