Redirecting standard I/O from within a program

You've come to this page because you've asked a question similar to

I know how one can use, for example, the redirection syntax in the 32-bit Command Interpreter to redirect a command's standard I/O before it is run. But how do I redirect the standard input/standard output/standard error of my program from within the program itself as it is running? Indeed, how do command interpreters themselves go about doing this before they invoke commands?

This is the Frequently Given Answer to that question.

APIs

This Frequently Given Answer makes a distinction between a file handle, which is an abstraction used at the OS/2, PC/MS/DR-DOS, and Win32 system APIs, and a file descriptor, which is an abstraction used at the POSIX API. It does this for clarity. But note that this is also an official distinction. Official POSIX, Unix, or Linux documentation, you'll find, only ever talks about file descriptors. The idea of file handles is a notion that, as far as can be determined, was invented by IBM/Microsoft for the PC/MS-DOS system API (and that then was carried forward into the OS/2 and Win32 system APIs).

One of the important points made in this Frequently Given Answer is that whilst the two are freely interchangeable with POSIX API libraries for OS/2 and PC/MS/DR-DOS, this isn't guaranteed to be the case, and most definitely is not the case for POSIX API libraries on other operating systems. The two are distinct and different abstractions.

There are three APIs at which the notion of "standard I/O" applies. How one redirects standard I/O depends, in part, from which levels one wishes to affect — what APIs one's program is using in order to perform I/O in the first place. These APIs are:

The Standard C library I/O streams.

Standard I/O at this level involves the stdin, stdout, and stderr streams. It is performed with Standard C library functions such as (amongst others) getc(), putc(), getchar(), putchar(), fgetc(), fputc(), printf(), and scanf().
The POSIX I/O API.

Standard I/O at this level involves the POSIX file descriptors numbered 0, 1, and 2. It is performed with POSIX API functions such as (amongst others) read(), write(), and ioctl().
The system API's I/O facilities.

Standard I/O at this level involves the underlying system API's I/O functionality. It varies according to the underlying target operating system for one's program. For examples:
- On OS/2 this involves file handles numbered 0, 1, and 2 and the DosRead(), DosWrite(), DosDevIOCtl(), and related API functions.
- On Win32 this involves file handles (without standard numbers) and the ReadFile(), WriteFile(), and related API functions.
- On PC/MS/DR-DOS this involves file handles numbered 0, 1, and 2 and various INT 21h functions, which the C/C++ implementation may have wrapped in C/C++-callable library functions such as _dos_read() and _dos_write().

Redirecting C streams

Some people will give you the very bad advice to open a new FILE object with fdopen() or fopen(), and then redirect the standard streams by assigning to their FILE objects thus:

FILE * f = fdopen(…) ;
if (!f) … ; // Handle errors.
*stdout = *f ;

Misguided purveyors of this very bad advice include P.S Sudheesh, Adam Sawicki, and Andrew Tucker (the latter having xyr bad advice published in the December 1997 edition of Windows Developer Journal, alas!).

This is bad advice, and bad programming. Do not do this. FILE objects are opaque types. You aren't supposed to muck around with their contents. That includes assigning to and from them. On most C and C++ implementations, at the very least you'll create a memory leak, as the pointers from the FILE object being overwritten to its internal buffers are lost. You risk a lot more than that, including lost I/O data, memory corruption, deadlocks caused by muddled-up semaphore and critical section use within multiple threads making C library I/O calls, and I/O data corruption at process termination by the library performing the same cleanup operations multiple times.

Do not dereference pointers to FILE objects, and don't expect such objects to be copyable or assignable. Opaque means opaque.

The Standard C library provides a function that one can use to cause the stdin, stdout, and stderr C streams to refer to other files: freopen(). It is used like this example of redirecting stdout:

FILE * f = freopen("newfile", "w", stdout) ;

For an explanation of the various parameters, see § 7.19.5.4 of the C standard, or your C/C++ implementation's Standard C library documentation.

The disadvantages to using freopen() on its own are severalfold:

It doesn't affect anything using either the POSIX API or system API I/O facilities.
It doesn't affect anything in the program using a different C runtime library (such as a third-party DLL, for example).
It doesn't enable one to redirect to anything that isn't in the file/directory namespace, such as a TCP/IP socket or an unnamed pipe for examples.
Such redirection is not inherited by child processes that are spawned with DosExec() (OS/2) or CreateProcess() (Win32); nor will it last across calls to execve() or one of the other "exec" calls (on POSIX systems).

These problems are addressed by redirecting at a lower layer, that the C library's I/O streams are layered on top of. The difficulty is that what they are layered on top of varies from implementation to implementation. On some implementations, they are layered on top of the POSIX API. This is (of course) the case for almost all C and C++ implementations for Unix and Linux systems. On other implementations, however, they are layered directly over the system API (when the POSIX API is not, in fact, the system API). This is the case for some C and C++ implementations for Win32, for example.

If one redirects at a lower layer, then one doesn't use freopen(). It's wholly unnecessary, and indeed it's counterproductive in that it negates the effect of the redirection at the lower layer. What one generally has to use are implementation-specific functions that allow one to tweak the linkage between the Standard C library's streams and the I/O API that they are layered on top of — more on which later — during or after performing the redirection at the lower layer.

Microsoft KnowledgeBase article 58667 states the falsehood that redirection using freopen() is reversible, by calling freopen() again. It is not. The Microsoft article is written based upon the overly simplistic notion that all programs always initially have their standard inputs, outputs, and errors attached to a console. They of course do not. It is also written based upon the outright false notion that opening a stream afresh yields the same stream as the original open stream. It of course does not — especially so in the cases of files (which could have been deleted in the interim, and at the very least will have different file pointers), sockets, and pipes. Reversibility requires some Standard C streams equivalent to the POSIX dup2() API function, to allow a stream to be saved to a copy and restored. Standard C provides no such thing.

Redirecting POSIX file descriptors

The POSIX API provides a function that one can use to cause file descriptors 0, 1, and 2 (standard input, standard output, and standard error) to refer to other files: dup2(). It is used like this example of redirecting standard error:

int fd = open(…) ; // … or a call to pipe() or socket()
if (0 > fd) … // Handle errors.
int r1 = dup2(fd, 2) ;
int r2 = close(fd) ;

For an explanation of the various parameters, see the POSIX IEEE 1003.1:2004 definition of dup2(), or your C/C++ implementation's POSIX API library documentation.

Unlike the case with freopen(), this mechanism is reversible, should one want to reverse it. One simply calls dup() to obtain a copy of the original file descriptor before performing the redirection. To reverse the redirection, one simply calls dup2() to copy the saved file descriptor back to the original standard file descriptor (and then of course close() to close the now-superfluous copy).

The POSIX API on Unix and its imitators

On operating systems such as Unix, Linux, GNU Hurd, and so forth, where the POSIX API is the system API, or at least is the lowest level API that one can call without breaking the interface contract between applications softwares and the operating system, this does everything that one needs to do. This is what Unix shells do in order to redirect the standard inputs/outputs/errors of processes that they fork().

There is nothing that one needs to do further in order to have the Standard C library C streams redirected. stdin, stdout, and stderr are at C library initialization associated with the POSIX file descriptors 0, 1, and 2, and any I/O through the C streams will just be directed to wherever those file descriptors actually lead.

There is only the one caveat that one need be aware of:

Strange things will happen if the Standard C library has data buffered in the internal I/O buffers of its stdin, stdout, and stderr streams. One must ensure that output is flushed using fflush() before one attempts to redirect standard output or standard error, otherwise the buffered data will end up being written to the wrong place.

Unfortunately, neither the C standard nor the POSIX standard provide mechanisms for flushing buffered-but-not-yet-read data from stdin before its underlying file descriptor is redirected. So one cannot do the same for redirecting standard input as for redirecting standard output or standard error.

There are several things that you'll find people doing, as a result of cargo-cult programming practices handed on from programmer to programmer, that are in fact unnecessary:

Some people will use the STDIN_FILENO, STDOUT_FILENO, and STDERR_FILENO macros, defined in the <unistd.h> header, instead of the numbers 0, 1, and 2 to refer to the standard file descriptors. This makes code more legible, and is a laudable practice. But it isn't necessary for portability. POSIX does guarantee that 0, 1, and 2 are always going to be the appropriate file descriptor numbers.
Sometimes people will use the fileno() macro to obtain the file descriptors from the stdin, stdout, and stderr streams. This isn't necessary unless one has already messed about with those streams via freopen(), which of course will associate different file descriptors with the C streams to the ones that they originally had. If one hasn't done that, then such indirection via fileno() is unnecessary. POSIX guarantees that at library initialization stdin, stdout, and stderr are associated with POSIX file descriptors 0, 1, and 2.

(Of course, if one has called freopen() to mess about with those streams, one has one's standard C streams I/O directed to places different to where one has one's standard POSIX file descriptor I/O directed. In such situations, one has to think carefully about what I/O functionality one actually wants to redirect in the first place.)
Sometimes people take the new file descriptor and pass it to fdopen() in order to obtain a new FILE object. This doesn't achieve anything useful. Aside from the fact that one may not assign that FILE object to another FILE object (such as, say, *stdout), meaning that one cannot redirect the Standard C streams in this way; it doesn't even affect I/O performed through the POSIX API read() and write() functions either. If one addresses the latter problem with dup2(), then fdopen() becomes entirely superfluous, because one already has FILE objects, standard ones, associated with the relevant file descriptors.

One doesn't need as many eggs as that in the pudding. The procedure is simple, and is as outlined afore.

The POSIX API on OS/2 and PC/MS/DR-DOS

On operating systems such as OS/2 and PC/MS/DR-DOS, whilst the POSIX API is not the system API, the POSIX API layer in the implementation's library is thin. The operating system's file handles have the same semantics as POSIX file descriptors in the relevant areas:

File handles are numbered from 0 upwards.
File handles 0, 1, and 2 are the conventional standard I/O handles.

As a consequence, the mapping from the one to the other is usually a direct one, with a file descriptor in the POSIX API being turned directly into a file handle at the system API. And calling dup2() will end up calling DosDupHandle() (OS/2) or invoking INT 21h/AX=45h (PC/MS/DR-DOS).

As such, the same is true for OS/2 and PC/MS/DR-DOS as for Unix and its imitators when it comes to redirecting I/O via the POSIX API. One just calls dup2() and closes the new file descriptor, as aforegiven, with the same simple caveat relating to buffering in the C streams.

And even though the POSIX API is an artefact of the implementation's library on such systems, because it's such a thin layer over the system API one does not have the worries about mixing code using multiple runtime libraries in a single process, or about handles not being inherited by spawned child processes.

The POSIX API on Win32

Things are very different with the POSIX API when it is layered on top of Win32. Win32 file handles do not support the semantics needed by the POSIX API for file descriptors:

They aren't numbered from 0 upwards.
They aren't numbered in increments of 1.
There are no standard (open) handle numbers.

To support the POSIX API, therefore, a Win32 C/C++ library has to provide a complex shim layer, that for starters maps POSIX file descriptor numbers onto Win32 file handles through a table maintained privately within the library. It is, in contrast to the situation for Unix, OS/2, and PC/MS/DR-DOS, a comparatively thick layer on top of the underlying Win32 API.

As a consequence, and for much the same reasons, redirecting POSIX file descriptors in a Win32 C or C++ implementation suffers from much the same problems as redirecting the Standard C streams using freopen():

It doesn't affect anything using the Win32 API I/O facilities.
It doesn't affect anything in the program using a different C runtime library (such as a third-party DLL, for example).
Such redirection is (unless the C library makes heroic efforts to try to encode its internal POSIX→Win32 file descriptor to file handle mapping in shared memory or environment variables) not inherited by child processes that are spawned with CreateProcess().

It suffers from specific additional problems of its own:

Not every C or C++ library layers the C streams on top of the POSIX API, to in turn be layered on top of the Win32 API. Several C and C++ implementations layer the C streams directly on top of the Win32 API. As a consequence of such a design, redirecting the POSIX API standard file descriptors has no effect at all on I/O performed via the C streams.

As with the case of the Standard C streams, these problems are addressed by redirecting at a lower layer, that the POSIX API's file descriptors is layered on top of, namely the Win32 API itself.

And as with the Standard C streams, what one generally has to use are implementation-specific functions that allow one to tweak the linkage between the POSIX API's file descriptors and the Win32 API that they are layered on top of — more on which later — during or after performing the redirection at the Win32 layer.

Redirecting System API file handles

In the cases of OS/2 and PC/MS/DR-DOS, redirection at the system API is effectively achieved by redirection at the POSIX API. System API file handles are directly interchangeable with POSIX API file descriptors, and one can call either dup2() or the equivalent underlying system API file handle duplication mechanism with equal effect.

In the case of Unix and its imitators, the system API is the POSIX API, and redirection at the system API is, again, effectively covered under redirection at the POSIX API.

This leaves Win32 as the sole case where redirection at the system API is a distinct case to be covered in its own right.

The Win32 System API

If one wants to avoid the aforementioned problems with freopen() and dup2() on Win32 C and C++ implementations, one has to redirect standard I/O at the Win32 system API level, also using implementation-specific functions that allow one to tweak the linkage between the Win32 API and the POSIX API and C streams that are layered on top of it.

Win32 has no concept of standard file handles. On Unix and its imitators, one can hardwire the standard file descriptor numbers 0, 1, and 2 (or the STDIN_FILENO, STDOUT_FILENO, and STDERR_FILENO macros from the <unistd.h> header) into an application. On OS/2 and PC/MS/DR-DOS, one can hardwire the file handle numbers 0, 1, and 2 into an application. There is no equivalent to this on Win32. There are no standard file handle numbers that one can hardwire into applications.

Instead, Win32 has what amounts to a small three-entry table of file handles, initialized by a process' parent in the call to CreateProcess() and queriable and modifiable within the process itself via GetStdHandle() and SetStdHandle(). Essentially, Win32 provides a process with a way of saying "Here are three file handle numbers, if anyone asks.".

Ironically, had this mechanism supported more than three handles, it would have made life a lot easier for C and C++ implementations wanting to provide a POSIX API over Win32. It would have provided a process-wide, compiler-neutral, language-neutral, inheritable from parent process to child process, mechanism for mapping from POSIX file descriptors to Win32 file handles. But since the Win32 table only supports three entries, C and C++ libraries have to roll their own tables, which are compiler-specific, not process-wide (when more than one runtime library is used within a process), and not easily inheritable from parent processes to child processes. Updating these tables involves calling C library functions that are not standardized, and that vary from implementation to implementation.

So when redirecting Win32 file handles, one has to separately and additionally inform the C library of the new Win32 file handles, using its implementation-specific mechanisms for modifying its internal POSIX file descriptor→Win32 file handle mapping table, and (if it is an implementation that doesn't layer the POSIX API between C streams and Win32) also modifying what Win32 file handles its C stream objects use.

With OpenWatcom C/C++, for example, this involves the implementation's _open_osfhandle() function:

HANDLE FileHandle = CreateFile(…) ;
BOOL r = SetStdHandle(STD_OUTPUT_HANDLE, FileHandle) ;
if (!r) … // Handle errors.
int fd = _open_osfhandle(FileHandle, O_WRONLY|O_TEXT) ;
if (0 > fd) … // Handle errors.
int r1 = dup2(fd, 1) ;
int r2 = close(fd) ;

The call to _open_osfhandle() returns a new POSIX file descriptor, with which one proceeds to redirect the POSIX standard file descriptors in the aforegiven fashion for POSIX, just as if one had obtained a new file descriptor using the POSIX open(), pipe(), or socket() API functions.

After the call to SetStdHandle(), any code in the process calling GetStdHandle(STD_OUTPUT_HANDLE) will obtain the new Win32 file handle to use. After the call to dup2(), any code in the process using the POSIX API with standard file descriptor number 1 or using the stdout C stream (since OpenWatcom layers the C streams on top of the POSIX API) will automatically use the new Win32 file handle.

Note that the original file handle is unaffected by the redirection, and still exists and remains open. The file descriptor is redirected, however. So one only needs to write the original file handle back in the table, but one needs to actually keep a saved duplicate file descriptor and redirect the file descriptor back.

Win32 does not have standard file handles, remember, and the "redirection" here is merely substituting the number of one open file handle for the number of another open file handle in a table of file handles, without changing the handles themselves. Only at the POSIX API level does a standard file descriptor actually get redirected to a different place, here.

This mechanism is, like redirecting POSIX file descriptors, reversible, should one want to reverse it. One simply calls GetStdHandle(STD_OUTPUT_HANDLE) to obtain the original file handle and dup() to obtain a copy of the original file descriptor, before performing the redirection. To reverse the redirection, one simply:

calls SetStdHandle(STD_OUTPUT_HANDLE, …) to copy the saved Win32 file handle back to the Win32 process-wide mapping table;
calls dup2() to copy the saved file descriptor back to the original standard file descriptor;
calls close() to close the now-superfluous saved file descriptor.

Again, the fact that Win32 does not provide a general-purpose mechanism with GetStdHandle() and SetStdHandle(), that can handle an arbitrary number of POSIX file descriptor→Win32 file handle mappings, is the underlying cause of the one problem that remains with redirecting Win32 file handles:

It doesn't affect anything in the program using a different C runtime library (such as a third-party DLL, for example).

If only the designers of Win32 had thought to provide a proper, general, API (instead of limited specialist functions that can only handle three mappings) which C/C++ library implementors could have used instead of having to roll their own, implementation-specific and runtime-library-instance-specific, mechanisms.

© Copyright 2010 Jonathan de Boyne Pollard. "Moral" rights asserted.
Permission is hereby granted to copy and to distribute this web page in its original, unmodified form as long as its last modification datestamp is preserved.