The last couple of days, coding has been genuinely rough: writing engineering code ended up messing up the repository, and writing algorithms had me tortured by one or two typos. In this state, I might as well lie in bed all day watching open courseware. Here is a simple note, recording how I got tripped up, plus a few small details.
On using the subprocess library
When writing server-side programs, we sometimes need to call third-party programs and obtain their output. If the program needs to run for a long time and keep producing output, and we also need to query that output frequently, then there are two approaches: either read the output in real time and return it directly to the client (letting the client "scroll"), or every time the client sends a query request, the server processes all output produced so far and returns it to the client. A somewhat hacky approach is to redirect output directly in the shell when invoking the program. That is certainly an option, but if we do not know in advance where to store the log file, or if, due to multithreading, we need multiple outputs at the same time, we then have to do things like generating random file names, which is not very clean or elegant. In practice, the subprocess library can solve this nicely.
We know subprocess is a modern library for handling inter-process message passing, intended to replace some older messy patterns—in other words, just use it. For I/O it has a few important parameters: redirecting stdin, stdout, and stderr. There are two good ways to use it: redirect to subprocess.PIPE, or redirect directly to a file handle. Essentially, the parameters specified here can be viewed as a buffer, so of course a file handle can be treated as a buffer; the effect is that all output is redirected straight into the file. But when reading, we will find that if we used PIPE, then we need to read from the file handle on the corresponding Popen object (that is, p.stdout.read()), rather than reading from PIPE, because PIPE is actually an I/O stream. You can basically think of it as a buffer-like concept: it only temporarily stores data, but to access it you still access the corresponding file handle on the Popen object.
But if we redirect output to a file handle, then we can read that file handle directly. Note: for this write-while-reading scenario, first you need to flush the write buffer, i.e., fid.flush—that goes without saying—but then, you must rewind the file pointer back to the beginning of the file! This is actually obvious, but it is easy to overlook, because after each write the pointer is already at the end of the file; at that point, trying to read data will read nothing, so you must rewind the pointer.
On the other hand, if we want to read output in real time and immediately print it or save it, then we have no choice but to use PIPE. At this time, whether the program behaves normally depends on whether the invoked program flushes its buffer in time. If it does, then we can repeatedly call readline(), or use iter on the readline function to access it as an iterable, to obtain output. Why is it so important for the called program to flush its buffer promptly? Because in Python's I/O model, if the data has not yet been output (i.e., it is still in the buffer), then a read operation will not return EOF; instead it will block the process. That means the program has to keep waiting on this unfinished read until it finally spits out everything it needs to output. In many cases, the program will not truly start outputting until it finishes running.
shell=True and su - root
This parameter of subprocess.Popen() indicates whether a shell needs to be invoked and the program executed within it. In general, it is not recommended to enable this parameter, because shell configurations differ greatly across operating systems, leading to serious platform dependency. But the problem is that some programs rely heavily on environment variables preset by the shell; if those variables are missing, the program may not run correctly at all. For example, for the web service I wrote that wraps a Coursera downloader, the issue is that if the downloader is not invoked in a shell environment, it obviously cannot find the executable path of aria2—this only just dawned on me. Many such bizarre problems are very hard to spot: it works fine when called in the command line! In this case, we have to first convert the original argument list into a command string, and then invoke a shell to execute that statement.
The same issue also exists in Linux's at and cron services. When executing commands, you can run into situations where it works in your shell, but once scheduled it either does nothing or fails. But when configuring these services, there is nowhere to specify that the command should run in some logged-in shell, so how do you handle it? The trick is to use the "-" parameter of the su command: during scheduled execution it will open a shell environment as if the specified user had logged in, and execute code within it. This becomes fully equivalent to manually executing the command in a shell, so it should not go wrong.
Automatic escaping in HTML templates
After writing some web projects, you will find that HTML templates are genuinely great for reducing workload and code redundancy. Template systems in different frameworks each have their own unique extension syntax; although they are all used in a rough-and-ready way, what we most need to pay attention to is when user variables passed into templates will be escaped, and how to disable escaping. For web.py's built-in template system, we use $ to reference a variable, and this process automatically escapes HTML symbols in the variable. If we do not need this feature, we can use $: to reference the variable, which disables automatic escaping.
This part is truly a trap. At first I thought something was wrong in the variable-passing part of my code—how could it cause the HTML not to be interpreted? In the end, after debugging in the Chrome console for a long time and checking the raw Response of the request, I finally realized that the variable contents had been escaped completely. No wonder it could not render!