Quick How-To for command-line Helpers

2022-05-17

Note: This is a "quick how-to". The instructions are subjectively reduced to a minimum, to keep focus and clarity.

Setup for the first time

[1] Unpack the archive:

unzip -o __ARCHIVE__.zip OR 7z x -y __ARCHIVE__.zip

[2] Enjoy.

The --help command line argument can be used.

Common information for all Helpers

• Some apps accept wildcard-patterns, some accept wildcard-expressions.
  • The syntax is described in Wildcard Patterns and Expressions describes the syntax.
• Same app can be used in scripts designed to work on Linux and on Windows.

Short description for each helper app

• cpmv ( Copy/Move )

  • like mkdir -p && cp or mkdir -p && mv

• lmt ( Last Modified Time )

  • like find -iname | grep | xargs | ls -t | head -1, faster
  • recursive search for files
  • accepts wildcard-patterns
  • print last modified time

• lsrec ( List Recursive )

  • like find -iname | grep | xargs | ls, faster
  • recursive search for files
  • accepts wildcard-patterns
  • print the list of files and summary

• repls ( Replace Strings )

  • like cat | awk replace > same_file, faster
  • by default a string is replaced/removed (case insensitive match)
  • uses wildcard expressions instead of regular expressions

• rmrec ( Remove Recursive )

  • like find -iname | grep | xargs | rm -r, faster with "force"
  • recursive remove files and directories
  • accepts wildcard-patterns
  • print the list of files and summary
  • by default it waits for user confirmation, unless "force" is used
  • non-aggressive, no chmod/chown with "force" argument

• runcaplog ( Run and Capture output in Log file )

  • like app | tee
  • captures stdout & stderr on same or separate files
  • successful capture depends on the redirects made by the shell or by the apps

• sortf ( Sort File )

  • like sort > same_file, faster
  • only ascii sort for now

• weg ( Wildcard Expressions Grep )

  • like grep, faster
  • uses wildcard expressions instead of regular expressions

• cats ( Cat and Squeeze blanks )

  • like cat -s
  • force new-line after last line

• realpathrec ( Real Path Recursive )

  • like realpath, no sensible change in output
  • it tries to show results in Windows, like in Linux

• touchrec ( Touch file Recursive )

  • like 'mkdir -p && touch`

• truncfile ( Truncate File to specified length )

  • like truncate

• waitkey ( Wait for a Key or until specified time elapses )

  • like bash read -n 1
  • wait for a key on stdin
  • return code is the (1st) key-code
  • optionally wait few seconds
  • successful operation depends on the terminal/console

Additional notes on capturing build-output

• Background:
  • It is not so easy as it should be with log messages from parallel builds.
  • Log messages may be mixed, because of the way OS & apps handle file flush.
  • Using performance optimizations, independent pipe channels ... these help up to a point.
  • The consumer should be faster than the producer.
  • The (pipe) buffers should not reach the full capacity.
• Why bother?
  • There can be weird issues caused by the build environment tools and/or setup.
  • Previous good logs can be compared with latest logs.
  • Logs from different build environments can be compared to find a build issue.
  • By forwarding messages on the original channel, the next processes in the pipe, including the IDE, receive what is expected.
  • Using a pipe3-like function in a script did not work as expected in many situations. Some messages were mixed when the system load was high.
• Good news:
  • Usually it is good enough to capture the output from a parallel build with a shell/script.
• Bad news:
  • Especially in hard times, when the system load is high, "Usually" is not the same as "Always", or at least not good enough.

Why 'runcaplog' exists?

• Want to:
  • Capture build logs from stderr and stdout channels on separate files.
  • Allow each message to pass on its own channel.
  • Minimize the likelihood of messages being mixed in the log.
• Solution: "runcaplog" helper.

Usual capture-output approaches:

• Combine stdout and stderr, then capture to a log file.
  • There is a chance that the messages will be mixed in the log.
  • The IDE or other processes may interpret some unwanted messages and may give unwanted results.
• Capture only one of stdout and stderr to a log file.
  • Only that channel can be analyzed later.

Usual capture-output implementations:

• If the IDE/build-apps you use accept stderr messages on stdout, then it could be ok with ( "$@" ) 2>&1 | tee log-file

• Else maybe the IDE/build-apps accept stdout messages on stderr, then it could be ok with

  ( "$@" ) 1>&2 | tee log-file

• Else capture without combining pipe channels. There are several possible variants, depending on the shell/applications used.

  `... use your-favorite-shell implementation here ...'

• Else capture one log per build action (compile,link,etc).

  • Less mixed messages & slower build overall & lots of independent logs.

• Else ( build in IDE and save the log when the issue appears ) use a tool which can handle independent stdout & stdin, and in the same time captures the output to one log file, like "runcaplog" or a similar dedicated app.

Additional notes regarding the license file

• With a "Default License", there is usually a small delay for non-free helper applications.
  • Delays can add up quickly when the application is run for batch processing, several times per second/minute.
• No delays during Happy Hours.
• You can purchase an "Active License" to eliminate delays.