"Your second-hand bookseller is second to none in the worth of the treasures he dispenses."
Leigh Hunt
Coming home to roost
This is the completion of a series on calling native code from high-level languages. Here is a description of the native library I'm calling in this series.
Apple has used several languages for its operating system and devices, most notably Objective-C and Swift. But I read a few years ago that Swift had found some adoption in data analysis and Big Data applications because of its expressiveness and streaming features. Swift has been released in open source, so there are implementations for Linux and Windows in addition to MacOS. I did an Advent of Code in Swift one year, and enjoyed it. To wrap up this project of calling native code from high-level languages I decided to give Swift a try.
Getting Started
The interface for calling native code from Swift has changed recently. The mechanism is the Swift Package Manager, but the changes have meant some older references are out of date. One example that gave me hope, even though it didn't work was this blog post: Wrapping C Libraries in Swift.
The example that got me going was directly from the Swift Documentation on the Swift Package Manager, particularly using system libraries to call native code.
As an Apple-original language, I wasn't sure how it would translate to Windows. I was fairly confident in its applicability to Linux, though, so that's where I started. That meant writing a command line application, instead of an app: those are Mac-only.
$ mkdir SwiftRMatrix
$ cd SwiftRMatrix
$ swift package init --type executable
$ tree .
.
├── Package.swift
└── Sources
└── SwiftRMatrix
└── SwiftRMatrix.swift
2 directories, 2 files
These commands set up a group of files and directories, the most important of which are Package.swift and Sources/SwiftRMatrix/SwiftRMatrix.swift. The latter is the entrypoint to the application, and the former is the directions for how to build the project. This is all that is needed to run "Hello, world!": you can do swift run at this point and see the message printed to the console.
Linking to native code is a matter of writing new modules and setting up dependencies among the modules in the project.
$ mkdir Sources/CRashunal
$ touch Sources/CRashunal/rashunal.h
$ touch Sources/CRashunal/module.modulemap
rashunal.h:
#import <rashunal.h>
module.modulemap:
module CRashunal [system] {
umbrella header "rashunal.h"
link "rashunal"
}
rashunal.h, which is distinct from the rashunal.h I wrote for the Rashunal project, is simply a transitive import to the native code, bringing all the declarations in the original rashunal.h into the Swift project. module.modulemap emphasizes this by saying that rashunal.h is an umbrella header, and that the code will link the rashunal library. At this point, CRashunal (the Swift project) can be imported into Swift code and used.
Package.swift:
// swift-tools-version: 6.2
// The swift-tools-version declares the minimum version of Swift required to build this package.
import PackageDescription
let package = Package(
name: "SwiftRMatrix",
dependencies: [],
targets: [
// Targets are the basic building blocks of a package, defining a module or a test suite.
// Targets can depend on other targets in this package and products from dependencies.
.systemLibrary(
name: "CRashunal"
),
.executableTarget(
name: "SwiftRMatrix",
dependencies: ["CRashunal"],
path: "Sources/SwiftRMatrix"
),
]
)
SwiftRMatrix.swift:
// The Swift Programming Language
// https://docs.swift.org/swift-book
import Foundation
@main
struct SwiftRMatrix {
public func run() throws {
let r: UnsafeMutablePointer = n_Rashunal(numericCast(1), numericCast(2))
print("{\(r.pointee.numerator),\(r.pointee.denominator)}")
}
}
I like that Swift distinguishes between mutable and immutable pointers (UnsafeMutablePointer and UnsafePointer), and uses generics to indicate what the pointer is to. Swift also has an OpaquePointer when the fields of a struct are not imported, like an RMatrix. I'll come back to that later. The pointee field to access the fields of the struct is an additional bonus.
ChatGPT pointed me to memory safety early on, so I learned quickly how to access the standard library on the different platforms. Swift recognizes C-like compiler directives, so accessing it was a simple matter of importing the right native libraries. For Windows, it's a part of the platform, so no special import is needed.
#if os(Linux)
import Glibc
#elseif os(Windows)
#elseif os(macOS)
import Darwin
#else
#error("Unsupported platform")
#endif
...
let r: UnsafeMutablePointer = n_Rashuna(numericCast(1), numericCast(2))
print("{\(r.pointee.numerator),\(r.pointee.denominator)}")
free(r)
And that's it, for code. The devil, of course, is in the compiling and linking.
A chain is only as strong as its weakest link
Swift Package Manager uses several sources to find libraries, but none of them seemed to match my particular use case. The closest was to make use of pkg-config. The more I read about it, the more it seemed to be an industry standard, and that Rashunal and RMatrix would benefit by taking advantage of it. So I broke my rule that I established earlier and decided to enhance the libraries.
Fortunately, it wasn't too painful. Telling Rashunal to write to pkg-config was only a few lines added to rashunal/CMakeLists.txt:
+set(PACKAGE_NAME rashunal)
+set(PACKAGE_VERSION 0.0.1)
+set(PACKAGE_DESC "Rational arithmetic library")
+set(PKGCONFIG_INSTALL_DIR "${CMAKE_INSTALL_LIBDIR}/pkgconfig")
+
+configure_file(
+ ${CMAKE_CURRENT_SOURCE_DIR}/rashunal.pc.in
+ ${CMAKE_CURRENT_BINARY_DIR}/${PACKAGE_NAME}.pc
+ @ONLY
+)
+
add_library(rashunal SHARED src/rashunal.c src/rashunal_util.c)
...
+install(
+ FILES ${CMAKE_CURRENT_BINARY_DIR}/rashunalConfig.cmake
+ DESTINATION lib/cmake/rashunal
+)
+
+install(
+ FILES ${CMAKE_CURRENT_BINARY_DIR}/${PACKAGE_NAME}.pc
+ DESTINATION ${PKGCONFIG_INSTALL_DIR}
)
The first block is toward the top of CMakeLists.txt, and the second is toward the bottom.
The configure_file directive needs a template for the pc file that will be written. The template has placeholders set of by '@' that will be filled in during the build process.
rashunal.pc.in:
prefix=@CMAKE_INSTALL_PREFIX@
exec_prefix=${prefix}
libdir=${exec_prefix}/@CMAKE_INSTALL_LIBDIR@
includedir=${prefix}/@CMAKE_INSTALL_INCLUDEDIR@
Name: @PACKAGE_NAME@
Description: @PACKAGE_DESC@
Version: @PACKAGE_VERSION@
Libs: -L${libdir} -l@PACKAGE_NAME@
Cflags: -I${includedir}
During installation the newly-written rashunal.pc file will be written to a platform-standard location on disk.
After making those changes, building, compiling, and installing, pkg-config was able to tell me something about the Rashunal library:
$ rm -rf build
$ mkdir build
$ cd build
$ cmake ..
$ make && sudo cmake --install .
$ ls /usr/local/lib/pkgconfig
rashunal.pc
$ cat /usr/local/lib/pkgconfig/rashunal.pc
prefix=/usr/local
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include
Name: rashunal
Description: Rational arithmetic library
Version: 0.0.1
Libs: -L${libdir} -lrashunal
Cflags: -I${includedir}
$ pkg-config --cflags rashunal
-I/usr/local/include
$ pkg-config --libs rashunal
-L/usr/local/lib -lrashunal
Notice the new command to install the project: apparently this is the more modern and more approved way to do it nowadays. The bash output means that the declarations of the Rashunal library can be found at /usr/local/include and the binaries at /usr/local lib.
Now the Swift Package Manager can be told just to consult pkg-config for the header and binary location of any system libraries it's attempting to build. It's not necessary, but the examples I saw recommended adding some suggestions for how to install Rashunal if it's not present. I haven't looked into what it takes to package a library for apt or brew, but I'm pretty sure this is how they are consumed:
Package.swift:
.systemLibrary(
name: "CRashunal",
pkgConfig: "rashunal",
providers: [
.apt(["rashunal"]),
.brew(["rashunal"]),
],
)
Then the Swift project could be built and run:
$ swift build
$ swift run SwiftRMatrix
{1,2}
And rinse and repeat for RMatrix. There is nothing new in building the RMatrix pkg-config files or linking to it from Swift, except for the dependency on Rashunal in the template for RMatrix:
rmatrix.pc.in
prefix=@CMAKE_INSTALL_PREFIX@
exec_prefix=${prefix}
libdir=${exec_prefix}/@CMAKE_INSTALL_LIBDIR@
includedir=${prefix}/@CMAKE_INSTALL_INCLUDEDIR@
Name: @PACKAGE_NAME@
Description: @PACKAGE_DESC@
Version: @PACKAGE_VERSION@
Requires: rashunal
Libs: -L${libdir} -l@PACKAGE_NAME@
Cflags: -I${includedir}
I started to look into removing that hardcoded dependency and getting it from the link libraries in CMakeLists.txt, but that quickly started to grow big and nasty, so I abandoned it. ChatGPT assured me that was common, especially for small projects.
Crossing the operating system ocean
Trying to do this on MacOS, I ran into my old nemesis SIP. Fortunately, the solution here was similar to the solution I followed there. The Swift command at /usr/bin/swift was protected by SIP, but the executable generated by the swift build command wasn't:
% swift build -Xlinker -rpath -Xlinker /usr/local/lib
% swift run .build/debug/SwiftRMatrix
{1,2}
What is astonishing is that, with one more testy exchange with ChatGPT, I also got it to work on Windows. I still don't understand what was the difference with Linux and MacOS or how this changed things on Windows, but I had to make an additional change to Rashunal's CMakeLists.txt and the cmake command to build RMatrix:
rashunal/CMakeLists.txt
if (WIN32)
set_target_properties(rashunal PROPERTIES
ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin"
RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin"
)
endif()
>cmake .. -G "NMake Makefiles" ^
More? -DCMAKE_BUILD_TYPE=Release ^
More? -DCMAKE_INSTALL_PREFIX=C:/Users/john.todd/local/rmatrix ^
More? -DCMAKE_PREFIX_PATH=C:/Users/john.todd/local/rashunal ^
More? -DCMAKE_C_FLAGS_RELEASE="/MD /O2 /DNDEBUG"
>nmake
>nmake install
Then the Swift application could be built and run from the command line, albeit with a few additional linker switches. This also needs to be done from a Powershell or DOS window with Admin rights because, even though it only changes the local project directory, it seems to write to a protected directory.
> swift build `
>> -Xcc -IC:/Users/john.todd/local/rashunal/include `
>> -Xcc -IC:/Users/john.todd/local/rmatrix/include `
>> -Xlinker /LIBPATH:C:/Users/john.todd/local/rashunal/lib `
>> -Xlinker /LIBPATH:C:/Users/john.todd/local/rmatrix/lib `
>> -Xlinker /DEFAULTLIB:rashunal.lib `
>> -Xlinker /DEFAULTLIB:rmatrix.lib `
>> -Xlinker /DEFAULTLIB:ucrt.lib
> ./.build/debug/SwiftRMatrix.exe
{1,2}
Cleaning up the guano
My last task was to abstract the native calls away from the main application. To do this I wrote a Models module that wrapped the native Rashunal, RMatrix, and Gauss Factorization structs.
Sources/Model/Model.swift
public class Rashunal: CustomStringConvertible {
var _rashunal: UnsafePointer
public init(_ numerator: Int, _ denominator: Int = 1) {
_rashunal = UnsafePointer(n_Rashunal(numericCast(numerator), numericCast(denominator)))
}
public init(_ data: [Int]) {
_rashunal = UnsafePointer(n_Rashunal(numericCast(data[0]), data.count > 1 ? numericCast(data[1]) : 1))
}
public var numerator: Int { Int(_rashunal.pointee.numerator) }
public var denominator: Int { Int(_rashunal.pointee.denominator) }
public var description: String {
return "{\(numerator),\(denominator)}"
}
deinit {
free(UnsafeMutablePointer(mutating: _rashunal))
}
}
What gets returned from the native n_Rashunal call is a Swift UnsafeMutablePointer. I wanted them to be immutable wherever possible, so I cast it to an UnsafePointer in both the constructors. Swift makes property definition and string representations easy and natural. The deinit method calls the native standard library's free method to release the native memory allocated by Rashunal. This makes cleanup and memory hygiene easy.
Sources/Model/Model.swift
public class RMatrix: CustomStringConvertible {
var _rmatrix: OpaquePointer
private init(_ rmatrix: OpaquePointer) {
_rmatrix = rmatrix
}
public init(_ data: [[[Int]]]) {
let height = data.count
let width = data.first!.count
let rashunals = data.flatMap {
row in row.map {
cell in n_Rashunal(numericCast(cell[0]), cell.count > 1 ? numericCast(cell[1]) : 1)
}
}
let ptrArray = UnsafeMutablePointer?>.allocate(capacity: rashunals.count)
for i in 0.. = RMatrix_get(_rmatrix, i, j)
let rep = "{\(cellPtr.pointee.numerator),\(cellPtr.pointee.denominator)}"
free(UnsafeMutablePointer(mutating: cellPtr))
return rep
}.joined(separator: " ") + " ]"
}.joined(separator: "\n")
}
deinit {
free_RMatrix(_rmatrix)
}
}
Unsurprisingly, RMatrix was the hardest of these to get right. The private constructor is used in the factor method as a convenience method to initialize a Swift RMatrix. The other constructor is used to initialize a matrix from the familiar 3D array of Ints. I get the height and width from the first two dimensions of the input array, then use the n_Rashunal method to construct a list of native Rashunal structs as UnsafeMutablePointer<CRashunal.Rashunal>s. As before, new_RMatrix expects an array of pointers to structs, but the rashunals array is in managed memory, not native memory. So I allocate and fill an array of pointers to the Rashunal structs in native memory. ChatGPT suggested I add the defer block in case new_RMatrix abends for any reason. Because the RMatrix struct is declared but not defined in rmatrix.h, what is automatically returned is an OpaquePointer, which is just fine with me.
Properties defer to the encapsulated _rmatrix pointer, and the string description method makes full use of Swift's stream processing capabilities. deinit calls the RMatrix library's free_RMatrix method.
After all that, factoring a matrix and the GaussFactorization struct are pretty routine.
Sources/Model/Model.swift
public struct GaussFactorization {
public var PInverse: RMatrix
public var Lower: RMatrix
public var Diagonal: RMatrix
public var Upper: RMatrix
public init(PInverse: RMatrix, Lower: RMatrix, Diagonal: RMatrix, Upper: RMatrix) {
self.PInverse = PInverse
self.Lower = Lower
self.Diagonal = Diagonal
self.Upper = Upper
}
}
public class RMatrix: CustomStringConvertible {
...
public func factor() -> GaussFactorization {
let gf = RMatrix_gelim(_rmatrix)!
let sgf = GaussFactorization(
PInverse: RMatrix(gf.pointee.pi),
Lower: RMatrix(gf.pointee.l),
Diagonal: RMatrix(gf.pointee.d),
Upper: RMatrix(gf.pointee.u)
)
free(gf)
return sgf
}
}
Calling the native method RMatrix_gelim returns a newly-allocated struct pointing to four newly-allocated matrices. The matrices are passed to the RMatrix constructor, so that the class takes responsibility for managing their memory. The native struct itself is freed by the RMatrix factor method before returning the Swift struct.
The driver class has no import of native code, and all the allocations look just like Swift objects.
import ArgumentParser
import Foundation
import Model
enum SwiftRMatrixError: Error {
case runtimeError(String)
}
@main
struct SwiftRMatrix: ParsableCommand {
@Option(help: "Specify the input file")
public var inputFile: String
public func run() throws {
let url = URL(fileURLWithPath: inputFile)
var inputText = ""
do {
inputText = try String(contentsOf: url, encoding: .utf8)
} catch {
throw SwiftRMatrixError.runtimeError("Error reading file [\(inputFile)]")
}
let data = inputText
.split(whereSeparator: \.isNewline)
.map { $0.trimmingCharacters(in: .whitespaces) }
.map { line in line.split(whereSeparator: { $0.isWhitespace })
.map { token in token.split(separator: "/").map { Int($0)! } }
}
let m = Model.RMatrix(data)
print("Input matrix:")
print(m)
let factor = m.factor()
print("Factors into:")
print("PInverse:")
print(factor.PInverse)
print("Lower:")
print(factor.Lower)
print("Diagonal:")
print(factor.Diagonal)
print("Upper:")
print(factor.Upper)
}
}
$ swift run SwiftRMatrix --input-file /home/john/workspace/rmatrix/driver/example.txt
[1/1] Planning build
Building for debugging...
[11/11] Linking SwiftRMatrix
Build of product 'SwiftRMatrix' complete! (1.17s)
Input matrix:
[ {-2,1} {1,3} {-3,4} ]
[ {6,1} {-1,1} {8,1} ]
[ {8,1} {3,2} {-7,1} ]
Factors into:
PInverse:
[ {1,1} {0,1} {0,1} ]
[ {0,1} {0,1} {1,1} ]
[ {0,1} {1,1} {0,1} ]
Lower:
[ {1,1} {0,1} {0,1} ]
[ {-3,1} {1,1} {0,1} ]
[ {-4,1} {0,1} {1,1} ]
Diagonal:
[ {-2,1} {0,1} {0,1} ]
[ {0,1} {17,6} {0,1} ]
[ {0,1} {0,1} {23,4} ]
Upper:
[ {1,1} {-1,6} {3,8} ]
[ {0,1} {1,1} {-60,17} ]
[ {0,1} {0,1} {1,1} ]
Reflection
Wow, that turned out a lot better than I expected. I thought this would be possible on Linux and MacOS. To be able to get it to work on Windows too was a pleasant surprise. I really like the Swift language: it is expressive and concise and makes really good use of streaming approaches. I hope I get to use it to make money sometime.
