Working Entirely with Python — A Concise Guide to Getting Started with F2PY

Preface

This post is the first in the planned “working entirely with Python” series. The goal of this series is to bring Python, a powerful tool, into plotting and computation in ocean science. I originally planned to base this semester’s marine factor computation assignments on F2PY, but there is not much Chinese material available, and the documentation is somewhat outdated. So after finishing the first (rather complex) assignment, I am writing a blog post to summarize the usage for future reference. Follow-up updates will be irregular; if you are interested, please visit my blog.

F2PY is an excellent glue module that merges Fortran programs capable of high-performance computing with the powerful and flexible Python language. As we know, MATLAB is essentially a toolbox that contains precompiled, high-performance low-level math modules, combined with a proprietary scripting language used to implement various higher-level functions, thereby providing a numerical computing solution for engineering and science.

In scientific computing, Python follows the same path. Python’s Numpy module provides a general-purpose multidimensional array object, with many built-in methods for operating on data. On top of that, Scipy provides high-performance low-level numerical computing modules. These are built on well-tested open-source libraries—for example, LAPACK for linear algebra—so performance is not bad.

Compared with the fully closed-source MATLAB, Python’s biggest advantage is that all its modules can be freely extended and redeveloped. So a well-designed module like the Array object, with common methods built in, can be used as a general storage container and as a bridge between programs—whether they are compute-intensive modules written in compiled languages or ordinary Python code. This is also why my Fortran assignments use the Array object: it can significantly improve the generality of the program and better integrate the functionality of self-developed modules with existing ones.

Example results

In general, Fortran code used for scientific computing is mostly written in the form of subroutines: it takes some array parameters, and the outputs are also array parameters. What F2PY does is bind Fortran arrays to Python Array objects, so that in Python code you can pass Python Array objects as input parameters to call a Fortran subroutine, and the returned values are also Array objects. Conversions such as data length and memory layout can all be done automatically. This means we no longer need to write a main program for the Fortran subroutine, because the invocation happens inside the Python runtime. After the data is processed by the Fortran module and returned as an Array object, we can continue to process it with functions in Numpy/Scipy.

Basic array input

subroutine dprod(x, y, n)  
integer, intent(in) :: n  
real(kind=8), intent(in) :: x(n)  
real(kind=8), intent(out) :: y  
y = 1.0  
do i=1, n  
y = y * x(i)  
end do  
end

The code above implements taking an Array object as input and computing the product of its elements, and it does not require specifying the array dimension or size at input time, so it is very convenient to use.
The calling method in Python is roughly as follows:

import test #import module  
a = linspace(1,10,10) #generate a row vector  
# output: array([  1.,   2.,   3.,   4.,   5.,   6.,   7.,   8.,   9.,  10.])  
test.dprod(a) #compute its product  
# output: 3628800.0

Operating on multidimensional arrays:

SUBROUTINE FOO(A,N,M)  
INTEGER N,M,I,J  
REAL(kind=8) A(N,M)  
!f2py intent(in,out,copy) a  
!f2py integer,intent(hide),depend(a) :: n=shape(a,0), m=shape(a,1)  
DO J=1,M  
A(1,J) = A(1,J) + 1D0  
ENDDO  
DO I=1,N  
A(I,1) = A(I,1) - 1D0  
ENDDO  
END

After adding the compilation directive lines above and compiling as described below, you get a Python module that can accept 2D Array objects; the calling process is not shown here.

A midterm tidal observation data harmonic analysis tool written with F2PY, and the entire implementation process.

Compilation command

Using F2PY on Linux should not be much of a problem, but using it on Windows is quite frustrating. First, you need a suitable compiler. At the moment I can only compile code using the gfortran compiler in Mingw, which means its location in the environment variables must be ahead of compilers like cygwin!

Compilation parameters verified to work:

f2py -c --fcompiler=gnu95 --compiler=mingw32 -m test test.f90

Here, the test after -m is the name of the compiled module. You can specify multiple filenames to compile different subroutines from multiple files into a single module. After compilation, usage is the same as a normal Python module: just import it and call functions as you like—though you still cannot pass arbitrary parameters, or you will trigger type exceptions. Of course, if one file references a subroutine defined in another file, you must include that other file during compilation; otherwise you will get unresolved external names.

If the error says it cannot find a certain .bat batch file, you need to set an environment variable:

VS90COMNTOOLS => %VS110COMNTOOLS% (for VS2012)
VS90COMNTOOLS => %VS120COMNTOOLS% (for VS2013)

Directive statements

F2PY needs to parse dependencies between input variables—for example, which variables are inputs, which are outputs, and which variables depend on the size of input arrays (i.e., Array objects; same below). This is also part of what makes it convenient. But for programs with more complex variable relationships, F2PY obviously cannot handle everything automatically, so you need to specify them manually.

Methods and differences

F2PY provides two ways to achieve this. One is to specify it using compilation directive statements in the code; the other is to generate a Signature File. You can understand this as having F2PY parse the Fortran semantics and generate a corresponding information file, which contains interface specifications in a Fortran-like syntax. We can manually modify this file so that the dependency relationships of variables match what we intend. Usage is as follows:

First, generate the corresponding signature file using the following command
f2py -m test -h test.pyf test.f90
Then modify the variable I/O attributes in it, and execute compilation
f2py -c --fcompiler=gnu95 --compiler=mingw32 test.pyf test.f90

Personally, I prefer the first approach, since it allows the code to be compiled directly without an extra step. The second method can be used to validate the first: if the module compiled from the code does not behave as expected, you can generate a signature file for the code and check what specific variable dependency relationships F2PY generated. In addition, for an already compiled module, you can directly inspect its input and output parameters in the same way you view help for ordinary Python functions: just add a question mark after the function name.

Specific syntax

F2PY compilation directive statements allow you to use the extended attributes of an F2PY signature file within Fortran77/90 source code to describe variable properties. This feature allows us to skip generating a signature file and apply F2PY directly to the Fortran source to build a Python module. The F2PY directive format is:
<comment char>f2py ...
For fixed-form Fortran code, the comment characters are “cC!#”, while for free-form it is “!”. For fixed-form code, <comment char> must appear in column 1; for free-form, the F2PY directive can appear anywhere in the file.
The compiler ignores everything after <comment char>f2py, but F2PY reads it like a normal code line. When F2PY finds a line containing an F2PY directive, it first replaces the directive with five spaces and then rereads the line, which means there must absolutely be no spaces after the comment character*!

The C expressions in directives (i.e., <init_expr> mentioned below) may include:

Standard C constructs;
Functions defined in math.h and Python.h;
Variables computed and initialized from the argument list based on the given dependencies;
The following C++ macros:
- rank(<name>) returns the number of dimensions of an array
- shape(<name>,<n>) returns the size of the n-th dimension of the array, where n starts from 0
- len(<name>) returns the array length
- size(<name>) returns the array size
- slen(<name>) returns the string length

Extended attributes and their syntax

In f90, the extended variable attributes include the following, which are used to adjust F2PY’s behavior.

Optional (optional)
The corresponding argument is moved to the end of the optional argument list. The default value of an optional argument is specified by <init_expr>. The default value must be a valid C expression; when <init_expr> is used, F2PY automatically sets the variable to optional.
All dimensions of an optional array argument must be bounded.
Required (required)
The corresponding argument is treated as required; this is the default. You only need to specify the required attribute when <init_expr> is used but you need to disable the automatic optional setting.
dimension(<arrayspec>)
The corresponding variable is treated as an array, whose dimensions are specified by <arrayspec>.
<arrayspec> is a comma-separated list of dimension bounds.
intent(<intentspec>)
This argument specifies the variable’s I/O attributes. It is a comma-separated expression containing the following attributes:
- in
  Treat the argument as an input variable, and the function cannot change its value.
- inout
  This attribute indicates the argument is both an input and an output variable—in other words, an in-place output variable. The variable must be a contiguous numeric array.
  Usually intent(inout) is not recommended; use intent(in,out) or the inplace attribute instead.
- inplace
  Similar to the previous attribute, but if the array type does not match exactly or the array is not contiguous, an in-place automatic conversion is performed to make the type match.
  Usually intent(inplace) is also not recommended, because it directly modifies the input variable. But if the input argument is only a temporary variable—for example, it is a slice of an array—then after that portion of memory is released it can lead to access to invalid memory.
- out
  Treat the variable as a return value and automatically append it to the end of the <returned variables> list. This attribute automatically sets intent(hide) unless other I/O attributes such as in or inout are explicitly set.
  By default, the memory layout for returned multidimensional arrays is column-major, i.e., Fortran order.
- hide
  This attribute removes the variable from the list of required or optional arguments. In general, intent(hide) is used only when intent(out) is used, or when <init_expr> can fully determine the variable’s value. For example:
  integer intent(hide),depend(a) :: n = len(a)
  real intent(in),dimension(n) :: a
- copy
  Ensures that the original contents of a variable with intent(in) are preserved. Often used together with intent(in,out), for example:
  !f2py intent(in,out,copy) a
- overwrite
  Indicates that the original contents of a variable with intent(in) may be modified by the function.
check([<C-booleanexpr>])
Evaluate <C-booleanexpr> to perform a consistency check on argument variables. If it returns False, an exception is raised. If this attribute is not explicitly used, F2PY will automatically generate some standard check statements, such as checking whether array sizes match, etc.
depend([<names>])
Declares that the argument variable with this attribute depends on the values of variables in the list <names>, i.e., specifies variable dependencies. This can be used, for example, to describe that an array size depends on some parameter, or that a scalar depends on the size of an input array. For instance, <init_expr> may use values of other arguments; using the information provided by the depend attribute, F2PY can ensure that all arguments are initialized in the correct order. If this attribute is not explicitly specified, F2PY will generate it automatically.
If you need to manually modify the depend attribute descriptions generated by F2PY, be careful not to break any relationships, and do not create circular dependencies, otherwise an error will be raised.

In addition, in Python there are only functions, not procedures (i.e., Fortran subroutines), and all non-array arguments are passed by value. So when Python calls a Fortran subroutine, variables marked as “intent(out)” are returned as function return values; and when more than one variable has this attribute, the return value is a tuple containing all returned variables.

Other details

Double-precision input

For some unknown reason, the F2PY version I use can only treat variables declared as real(kind=8) as double precision, and does not accept other forms. For a detailed analysis, see this Q&A on StackOverflow here.

Automatic inference of array sizes

Also for some unknown reason, F2PY forces input parameters that can be inferred from array sizes to be automatically converted into hidden parameters (i.e., not included in the input parameter list). If you do not need such automatic optimization, in theory you can solve it by modifying the signature file and recompiling.