Why do humans like old things?
Dr. Noonien Soong, Star Trek the Next Generation
Although I haven't actually done it very much, I've always been fascinated by the idea of calling into old code from modern applications. Who knows what value is locked away in those old libraries? Graphics, matrix calculations, statistics, quantum mechanical calculations, etc. I want to be able to do it all!
In reality, old code is probably dusty, unmaintained, and harder to use than modern code. I'm still interested in being able to access it.
As discussed in the introduction to this series, I wrote a small library to do matrix calculations on rational numbers so that I could focus on the calculations without worrying about data loss or errors due to rounding. This post is about calling into it from Java via the Java Native Interface (JNI).
Starting point
To get a basic education I started with Baeldung's Tutorial on JNI. This gave a few basic examples of how to write the Java code, compile it, generate the C header file, write and compile the implementation of that, and call it all together. In particular, the Using Objects and Calling Java Methods From Native Code section was a good introduction to generating Java objects from the native side of the world.
I did most of this work in an Ubuntu image in a WSL on a Windows machine. I used Java 11 SDK for the Java compilation steps.
Calling RMatrix from Java, creating Java objects from the results
I wrote some simple Java classes as counterparts of the C structs. Then I wrote a simple driver to create small matrix, call the Gauss Factorization method, and display the U matrix. To call the native method I decided to pass a three-dimensional integer array. The first two dimensions represent the height and width of the matrix. The third dimension is either a one- or two-element array representing the numerator and denominator of a rational number, with a one-dimensional array representing a denominator of 1. This matches well with the behavior of String.split("/").
package org.jtodd.jni;
public class JRashunal {
private int numerator;
private int denominator;
public JRashunal(int numerator, int denominator) {
this.numerator = numerator;
this.denominator = denominator;
}
@Override
public String toString() {
if (denominator == 1) {
return String.format("{%d}", numerator);
} else {
return String.format("{%d/%d}", numerator, denominator);
}
}
}
package org.jtodd.jni;
public class JRashunalMatrix {
private int height;
private int width;
private JRashunal[] data;
public JRashunalMatrix(int height, int width, JRashunal[] data) {
this.height = height;
this.width = width;
this.data = data;
}
@Override
public String toString() {
StringBuilder builder = new StringBuilder();
for (int i = 0; i < height; ++i) {
builder.append("[ ");
for (int j = 0; j < width; ++j) {
builder.append(data[i * width + j]);
builder.append(" ");
}
builder.append("]\n");
}
return builder.toString();
}
}
package org.jtodd.jni;
public class RMatrixJNI {
static {
System.loadLibrary("jnirmatrix");
}
public static void main(String[] args) {
RMatrixJNI app = new RMatrixJNI();
int data[][][] = {
{ { 1 }, { 2 }, { 3, 2 }, },
{ { 4, 3 }, { 5 }, { 6 }, },
};
JRashunalMatrix u = app.factor(data);
System.out.println(u);
}
private native JRashunalMatrix factor(int data[][][]);
}
The Java code is compiled and the C header file is generated in the same step:
$ javac -cp build -h build -d build RMatrixJNI.java JRashunal.java JRashunalMatrix.java
Other blogs, including Baeldung's, show you what a JNI header file looks like, so I won't copy it all here. The most important line is the declaration of the method defined in the Java class:
JNIEXPORT jobject JNICALL Java_org_jtodd_jni_RMatrixJNI_factor
(JNIEnv *, jobject, jobjectArray);
This is the method that you have to implement in the code you write. Include the header file generated by the Java compiler, as well as the Rashunal and RMatrix libraries. I named the C file the same as the header file generated by the compiler.
#include "rashunal.h"
#include "rmatrix.h"
#include "org_jtodd_jni_RMatrixJNI.h"
JNIEXPORT jobject JNICALL Java_org_jtodd_jni_RMatrixJNI_factor (JNIEnv *env, jobject thisObject, jobjectArray jdata)
{
...
}
After I got this far, Baeldung couldn't help me much anymore. I turned to the full list of functions defined in the JNI specification. This let me get the dimensions of the Java array and allocate the array of C Rashunals:
long height = (long)(*env)->GetArrayLength(env, jdata);
jarray first_row = (*env)->GetObjectArrayElement(env, jdata, 0);
long width = (long)(*env)->GetArrayLength(env, first_row);
size_t total = height * width;
Rashunal **data = malloc(sizeof(Rashunal *) * total);
It took some fiddling, but then I figured out how to get data from the elements of the 2D array, create C Rashunals, create the C RMatrix, and factor it:
for (size_t i = 0; i < total; ++i) {
size_t row_index = i / width;
size_t col_index = i % width;
jarray row = (*env)->GetObjectArrayElement(env, jdata, row_index);
jarray jel = (*env)->GetObjectArrayElement(env, row, col_index);
long el_count = (long)(*env)->GetArrayLength(env, jel);
jint *el = (*env)->GetIntArrayElements(env, jel, JNI_FALSE);
int numerator = (int)el[0];
int denominator = el_count == 1 ? 1 : (int)el[1];
data[i] = n_Rashunal(numerator, denominator);
}
RMatrix *m = new_RMatrix(height, width, data);
Gauss_Factorization *f = RMatrix_gelim(m);
const RMatrix *u = f->u;
size_t u_height = RMatrix_height(u);
size_t u_width = RMatrix_width(u);
The really tricky part was finding the Java class and constructor definitions from within the native code. The JNI uses something called descriptors to refer to primitives and objects:
- The descriptors for primitives are single letters:
Ifor integer,Zfor boolean, etc. - The descriptor for a class is the fully-qualified class name, preceded by an L and trailed by a semicolon:
Lorg/jtodd/jni/JRashunal;. - The descriptor for an array is the primitive/class descriptor preceded by an opening bracket:
[I,[Lorg/jtodd/jni/JRashunal;. Multidimensional arrays add an opening bracket for each dimension of the array. - The descriptor for a method is the argument descriptors in parentheses, followed by the descriptor of the return value.
- You can see an example of this in the header file generated by the Java compiler for the Java class: it accepts a three-dimensional array of integers and returns a JRashunalMatrix, so the signature is
([[[I)Lorg/jtodd/jni/JRashunalMatrix;. - If a method has multiple arguments, the descriptors are concatenated with no delimiter. This caused me a lot of grief because I couldn't find any documentation about it. ChatGPT finally gave me the clue to this. It also told me a handy tool to find the method signature of a compiled class:
javap -s -p fully.qualified.ClassName.
V):
jclass j_rashunal_class = (*env)->FindClass(env, "org/jtodd/jni/JRashunal");
jclass j_rmatrix_class = (*env)->FindClass(env, "org/jtodd/jni/JRashunalMatrix");
jmethodID j_rashunal_constructor = (*env)->GetMethodID(env, j_rashunal_class, "", "(II)V");
jmethodID j_rmatrix_constructor = (*env)->GetMethodID(env, j_rmatrix_class, "", "(II[Lorg/jtodd/jni/JRashunal;)V");
(Those II's look like the Roman numeral 2!)
That was the hard part. Although the syntax is ugly, allocating and populating an array of JRashunals and creating a JRashunalMatrix was pretty straightforward:
jobjectArray j_rashunal_data = (*env)->NewObjectArray(env, u_height * u_width, j_rashunal_class, NULL);
for (size_t i = 0; i < total; ++i) {
const Rashunal *r = RMatrix_get(u, i / width + 1, i % width + 1);
jobject j_rashunal = (*env)->NewObject(env, j_rashunal_class, j_rashunal_constructor, r->numerator, r->denominator);
(*env)->SetObjectArrayElement(env, j_rashunal_data, i, j_rashunal);
free((Rashunal *)r);
}
jobject j_rmatrix = (*env)->NewObject(env, j_rmatrix_class, j_rmatrix_constructor, RMatrix_height(u), RMatrix_width(u), j_rashunal_data);
Compiling, linking, and running
Up to now I've assumed you understand the basics of C syntax, compiling, linking, and running. I won't assume that for the rest of this because it got pretty tricky and took me a while to figure it out.
I've laid out my project like this:
$ tree .
.
├── JRashunal.java
├── JRashunalMatrix.java
├── RMatrixJNI.java
├── build
│ ├── all generated and compiled code
└── org_jtodd_jni_RMatrixJNI.c
4 directories, 31 files
I set `JAVA_HOME` to the root of the Java 11 SDK I'm using. To compile the C file:
$ echo $JAVA_HOME
/usr/lib/jvm/java-11-openjdk-amd64
$ cc -c -fPIC \
-Ibuild \
-I${JAVA_HOME}/include \
-I${JAVA_HOME}/include/linux \
org_jtodd_jni_RMatrixJNI.c \
-o build/org_jtodd_jni_RMatrixJNI.o
Adjust the includes to find the JNI header files for your platform. If you installed Rashunal and RMatrix to a recognized location (/usr/local/include for me) the compiler should find them on its own. If not, add includes to them as well.
$ cc -shared -fPIC -o build/libjnirmatrix.so build/org_jtodd_jni_RMatrixJNI.o -L/usr/local/lib -lrashunal -lrmatrix -lc
To create the shared library you have to link in the Rashunal and RMatrix libraries, hence the additional link location and link switches.
$ LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH java -cp build \
-Djava.library.path=/home/john/workspace/JavaJNI/build org.jtodd.jni.RMatrixJNI
[ {1} {2} {3/2} ]
[ {0} {1} {12/7} ]
This is the tricky one. Since we created a shared library (libjnirmatrix.so), we need to provide the runtime the path to the linked Rashunal and RMatrix libraries. This isn't done through the java.library.path variable; that tells the JVM where to find the JNI header file. You need the system-specific load path to tell the system runtime (not the JVM runtime) where to find the linked libraries. On Linux and MacOS that's the LD_LIBRARY_PATH variable. Thanks again, ChatGPT!
Whew, that's a lot of work! And who wants to type those absurdly long CLI commands?
Doing it in a modern build system: Gradle
I'm continuing in my Ubuntu WSL shell with Java 11 and Gradle 8.3.
$ mkdir jrmatrix
$ gradle init
# I chose a basic project with Groovy as the DSL and the new APIs
Apparently Gradle's Java and native plugins don't play nicely in the same project, so the first thing I did was separate the project into app (Java) and native (C) subprojects. All of the automatically-generated files could stay the way they were. I just needed to make a small change to settings.gradle:
rootProject.name = 'jrmatrix'
include('app', 'native')
Then I made folders for each subproject.
In the app subproject I created the typical Java folder structure:
$ tree .
.
├── build.gradle
└── src
└── main
└── java
└── org
└── jtodd
└── jni
├── JRashunal.java
├── JRashunalMatrix.java
├── RMatrixJNI.java
└── jrmatrix
└── App.java
26 directories, 11 files
The build.gradle file in app needed switches to tell the Java compiler to generate the JNI header file and where to put it. This is no longer a separate command (javah), but is an additional switch on the javac command. In addition, I wanted the run task to depend on the native compilation and linking steps I'll describe later in the native subproject.
plugins {
id 'application'
id 'java'
}
tasks.named("compileJava") {
def headerDir = file("${buildDir}/generated/jni")
options.compilerArgs += ["-h", headerDir.absolutePath]
outputs.dir(headerDir)
}
application {
mainClass = 'org.jtodd.jni.jrmatrix.App'
applicationDefaultJvmArgs = [
"-Djava.library.path=" + project(":native").layout.buildDirectory.dir("libs/shared").get().asFile.absolutePath
]
}
tasks.named("run") {
dependsOn project(":native").tasks.named("linkJni")
}
Most of the fun was in the native project. I set it up with a similar folder structure to Java projects:
$ tree .
.
├── build.gradle
└── src
└── main
└── c
└── org_jtodd_jni_RMatrixJNI.c
6 directories, 4 files
Gradle has a cpp-library plugin, but it seems tailored just to C++, not to C. There is also a c-library plugin, but that seems not to be bundled with Gradle 8.3, so I decided to skip it. The other alternative was a bare-bones c plugin, which apparently is pretty basic. Much of the code will look similar to what I did earlier at the command line.
Like I did there, I had to write separate steps to compile and link the implementation of the JNI header file with the Rashunal and RMatrix libraries. After a couple of refactorings to pull out common definitions of the includes and not hardcoding the names of C source files I wound up with this:
apply plugin: 'c'
def jniHeaders = { project(":app").layout.buildDirectory.dir("generated/jni").get().asFile }
def jvmHome = { file(System.getenv("JAVA_HOME")) }
def outputDir = { file("$buildDir/libs/shared") }
def osSettings = {
def os = org.gradle.nativeplatform.platform.internal.DefaultNativePlatform.currentOperatingSystem
def baseInclude = new File(jvmHome(), "/include")
def includeOS
def libName
if (os.isLinux()) {
includeOS = new File(jvmHome(), "/include/linux")
libName = "libjnirmatrix.so"
} else if (os.isMacOsX()) {
includeOS = new File(jvmHome(), "/include/darwin")
libName = "libjnirmatrix.dylib"
} else if (os.isWindows()) {
includeOS = new File(jvmHome(), "/include/win32")
libName = "jnirmatrix.dll"
} else if (os.isFreeBSD()) {
includeOS = new File(jvmHome(), "/include/freebsd")
libName = "libjnirmatrix.so"
} else {
throw new GradleException("Unsupported OS: $os")
}
[baseInclude, includeOS, libName]
}
def sourceDir = file("src/main/c")
def cSources = fileTree(dir: sourceDir, include: "**/*.c")
def objectFiles = cSources.files.collect { file ->
new File(outputDir(), file.name.replaceAll(/\.c$/, ".o")).absolutePath
}
tasks.register('compileJni', Exec) {
dependsOn project(":app").tasks.named("compileJava")
outputs.dir outputDir()
doFirst { outputDir().mkdirs() }
def (baseInclude, includeOS, _) = osSettings()
def compileArgs = cSources.files.collect { file ->
[
'-c',
'-fPIC',
'-I', jniHeaders().absolutePath,
'-I', baseInclude.absolutePath,
'-I', includeOS.absolutePath,
file.absolutePath,
'-o', new File(outputDir(), file.name.replaceAll(/\.c$/, ".o")).absolutePath
]
}.flatten()
commandLine 'gcc', *compileArgs
}
tasks.register('linkJni', Exec) {
dependsOn tasks.named("compileJni")
outputs.dir outputDir()
doFirst { outputDir().mkdirs() }
def (baseInclude, includeOS, libName) = osSettings()
commandLine 'gcc',
'-shared',
'-fPIC',
'-o', new File(outputDir(), libName).absolutePath,
*objectFiles,
'-I', jniHeaders().absolutePath,
'-I', baseInclude.absolutePath,
'-I', includeOS.absolutePath,
'-L', '/usr/local/lib',
'-l', 'rashunal',
'-l', 'rmatrix',
'-Wl,-rpath,/usr/local/lib'
}
tasks.named('build') {
dependsOn tasks.named('compileJni')
dependsOn tasks.named('linkJni')
}
Gradle subprojects have references to each other, so this library can get references to app's output directory to reference the JNI header file. The compileJni task is set to depend on app's compileJava task, and native's build task is set to depend on the compileJni and linkJni tasks defined in this file.
This worked if I explicitly called app's compileJava task and native's build task, but it failed after a clean task. It turned out Java's compile task wouldn't detect the deletion of the JNI header file as a change that required rebuilding, so I added the build directory as an output to the task (outputs.dir(headerDir)). Thus deleting that file (or cleaning the project) caused recompilation and rebuilding.
The nice thing is that this runs with a single command now (`./gradlew run`). Much nicer than entering all the command line commands by hand!
Reflection
As expected, this works but is very fragile. Particularly calling Java code from the native code depends on knowledge of the class and method signatures. If those change in the Java code, the project will compile and start just fine, but blow up pretty explosively and nastily with unclear explanations at runtime.
I was surprised by Gradle's basic tooling for C projects. I thought there would be more help than paralleling the command line so closely. I'll have to look into the `c-library` plugin to see if it offers any more help. I'm also surprised by how few blogs and Stack Overflow posts I found about this: apparently this isn't something very many people do (or live to tell the tale!).
