Execute and Control System Processes
Tools to run system processes in the background, read their standard output and error and kill them.
processx can poll the standard output and error of a single process, or multiple processes, using the operating system’s polling and waiting facilities, with a timeout.
Note: the following external commands are usually present in macOS and Linux systems, but not necessarily on Windows. We will also use the
pxcommand line tool (
px.exeon Windows), that is a very simple program that can produce output to
stderr, with the specified timings.
#>  "/Users/gaborcsardi/r_pkgs/processx/bin/px"
run() function runs an external command. It requires a single command, and a character vector of arguments. You don’t need to quote the command or the arguments, as they are passed directly to the operating system, without an intermediate shell.
#> $status #>  0 #> #> $stdout #>  "Hello R!\n" #> #> $stderr #>  "" #> #> $timeout #>  FALSE
Short summary of the
px binary we are using extensively below:
#> Usage: px [command arg] [command arg] ... #> #> Commands: #> sleep <seconds> -- sleep for a number os seconds #> out <string> -- print string to stdout #> err <string> -- print string to stderr #> outln <string> -- print string to stdout, add newline #> errln <string> -- print string to stderr, add newline #> errflush -- flush stderr stream #> cat <filename> -- print file to stdout #> return <exitcode> -- return with exitcode #> write <fd> <string> -- write to file descriptor #> echo <fd1> <fd2> <nbytes> -- echo from fd to another fd #> getenv <var> -- environment variable to stdout
Note: From version 3.0.1, processx does not let you specify a full shell command line, as this involves starting a grandchild process from the child process, and it is difficult to clean up the grandchild process when the child process is killed. The user can still start a shell (
cmd.exe) directly of course, and then proper cleanup is the user’s responsibility.
run() throws an error if the process exits with a non-zero status code. To avoid this, specify
error_on_status = FALSE:
#> $status #>  2 #> #> $stdout #>  "oh no!" #> #> $stderr #>  "" #> #> $timeout #>  FALSE
To show the output of the process on the screen, use the
echo argument. Note that the order of
stderr lines may be incorrect, because they are coming from two different connections.
#> out #> out again #> err
If you have a terminal that support ANSI colors, then the standard error output is shown in red.
The standard output and error are still included in the result of the
#> $status #>  0 #> #> $stdout #>  "out\nout again\n" #> #> $stderr #>  "err\n" #> #> $timeout #>  FALSE
run() is different from
system(), and it always shows the output of the process on R’s proper standard output, instead of writing to the terminal directly. This means for example that you can capture the output with
capture.output() or use
#>  "CODE_OF_CONDUCT.md" "DESCRIPTION" "LICENSE" #>  "LICENSE.md" "Makefile" "NAMESPACE" #>  "NEWS.md" "R" "README.Rmd" #>  "README.md" "_pkgdown.yml" "appveyor.yml" #>  "docs" "inst" "man" #>  "processx.Rproj" "r-packages" "src" #>  "tests"
spinner option of
run() puts a calming spinner to the terminal while the background program is running. The spinner is always shown in the first character of the last line, so you can make it work nicely with the regular output of the background process if you like. E.g. try this in your R terminal:
result <- run(px, c("out", " foo", "sleep", "1", "out", "\r bar", "sleep", "1", "out", "\rX foobar\n"), echo = TRUE, spinner = TRUE)
run() can call an R function for each line of the standard output or error of the process, just supply the
stdout_line_callback or the
stderr_line_callback arguments. The callback functions take two arguments, the first one is a character scalar, the output line. The second one is the
process object that represents the background process. (See more below about
process objects.) You can manipulate this object in the callback, if you want. For example you can kill it in response to an error or some text on the standard output:
#> Got: this #> Got: that #> Got: done #> Got: still here
#> $status #>  -9 #> #> $stdout #>  "this\nthat\ndone\nstill here\n" #> #> $stderr #>  "" #> #> $timeout #>  FALSE
Keep in mind, that while the R callback is running, the background process is not stopped, it is also running. In the previous example, whether
still here is printed or not depends on the scheduling of the R process and the background process by the OS. Typically, it is printed, because the R callback takes a while to run.
In addition to the line-oriented callbacks, the
stderr_callback arguments can specify callback functions that are called with output chunks instead of single lines. A chunk may contain multiple lines (separated by
\r\n), or even incomplete lines.
If you need better control over possibly multiple background processes, then you can use the R6
process class directly.
To start a new background process, create a new instance of the
A process can be killed via the
#>  TRUE
#>  TRUE
#>  FALSE
Note that processes are finalized (and killed) automatically if the corresponding
process object goes out of scope, as soon as the object is garbage collected by R:
#> used (Mb) gc trigger (Mb) limit (Mb) max used (Mb) #> Ncells 428443 22.9 908452 48.6 NA 631340 33.8 #> Vcells 867615 6.7 8388608 64.0 16384 1769037 13.5
Here, the direct call to the garbage collector kills the
sleep process as well. See the
cleanup option if you want to avoid this behavior.
By default the standard output and error of the processes are ignored. You can set the
stderr constructor arguments to a file name, and then they are redirected there, or to
"|", and then processx creates connections to them. (Note that starting from processx 3.0.0 these connections are not regular R connections, because the public R connection API was retroactively removed from R.)
read_error_lines() methods can be used to read complete lines from the standard output or error connections. They work similarly to the
readLines() base R function.
Note, that the connections have a buffer, which can fill up, if R does not read out the output, and then the process will stop, until R reads the connection and the buffer is freed.
Always make sure that you read out the standard output and/or error of the pipes, otherwise the background process will stop running!
If you don’t need the standard output or error any more, you can also close it, like this:
Note that the connections used for reading the output and error streams are non-blocking, so the read functions will return immediately, even if there is no text to read from them. If you want to make sure that there is data available to read, you need to poll, see below.
The standard R way to query the end of the stream for a non-blocking connection, is to use the
isIncomplete() function. After a read attempt, this function returns
FALSE if the connection has surely no more data. (If the read attempt returns no data, but
TRUE, then the connection might deliver more data in the future.
is_incomplete_error() functions work similarly for
poll_io() method waits for data on the standard output and/or error of a process. It will return if any of the following events happen:
For example the following code waits about a second for output.
#> output error process #> "ready" "nopipe" "nopipe"
#>  "kuku"
If you need to manage multiple background processes, and need to wait for output from all of them, processx defines a
poll() function that does just that. It is similar to the
poll_io() method, but it takes multiple process objects, and returns as soon as one of them have data on standard output or error, or a timeout expires. Here is an example:
#> $p1 #> output error process #> "timeout" "nopipe" "nopipe" #> #> $p2 #> output error process #> "nopipe" "timeout" "nopipe"
#> $p1 #> output error process #> "ready" "nopipe" "nopipe" #> #> $p2 #> output error process #> "nopipe" "silent" "nopipe"
#>  "output"
#> $p1 #> output error process #> "closed" "nopipe" "nopipe" #> #> $p2 #> output error process #> "nopipe" "ready" "nopipe"
#>  "error"
As seen before,
is_alive() checks if a process is running. The
wait() method can be used to wait until it has finished (or a specified timeout expires).. E.g. in the following code
wait() needs to wait about 2 seconds for the
px command to finish.
#>  TRUE
#>  "2018-07-24 19:57:33 BST"
#>  "2018-07-24 19:57:35 BST"
It is safe to call
wait() multiple times:
After a process has finished, its exit status can be queried via the
get_exit_status() method. If the process is still running, then this method returns
#>  0
In general, mixing processx (via callr or not) and parallel works fine. If you use parallel’s ‘fork’ clusters, e.g. via
parallel::mcparallel(), then you might see two issues. One is that processx will not be able to determine the exit status of some processx processes. This is because the status is read out by parallel, and processx will set it to
NA. The other one is that parallel might complain that it could not clean up some subprocesses. This is not an error, and it is harmless, but it does hold up R for about 10 seconds, before parallel gives up. To work around this, you can set the
PROCESSX_NOTIFY_OLD_SIGCHLD environment variable to a non-empty value, before you load processx. This behavior might be the default in the future.
Errors are typically signalled via non-zero exits statuses. The processx constructor fails if the external program cannot be started, but it does not deal with errors that happen after the program has successfully started running.
#> Error in rethrow_call(c_processx_exec, command, c(command, args), stdin, : cannot start processx process 'nonexistant-command-for-sure' (system error 2, No such file or directory) @unix/processx.c:584 (processx_exec)
#>  5
Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.