Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Shared objects on Linux
26-10-2011, 07:46 PM
Post: #1
Shared objects on Linux
Disclaimer: My knowledge of Linux is sketchy, and if the following contains inaccuracies or misunderstandings, I would appreciate any corrections and clarifications.

I've been having a bit of difficulty building libohNetJni.so for Linux (Debian). The problem is that it depends on libohNet.so, and the path for this dependency needs to be hard-wired into the libohNetJni.so file.

There's no equivalent to this on Windows, which is happy to load ohNet.dll and ohNetJni.dll from any locations at runtime.

After considerable research and experimentation, I've concluded that I should adopt the Linux/Debian practice of assigning a fixed location to every package that's installed and hard-wiring this location into any other shared objects that depend on the package. As a temporary measure, I've used a location of /usr/local/bin/ohnet when linking the ohNet shared objects and everything is working OK if I place the ohNet runtime files in this location.

Is this use of a fixed installation location the correct practice for Linux? If so, what location should I be using for the libohNet.so and libohNetJni.so files?
Find all posts by this user
27-10-2011, 10:27 AM
Post: #2
RE: Shared objects on Linux
Hi, I'm Andrew and I work with Simon on ohNet. I mostly work with C# code running in Mono, but I've looked into this a little, and I'll try to explain my understanding. There are probably better Linux/Posix experts out there who can explain things better.

I believe you're right about use of a fixed installation location on Posix systems. The normal way this is handled is to have a "configure" step before building, which includes setting this fixed installation location to be used during compilation, generally by setting PREFIX. ohNet doesn't have this step, and I think it's currently tricky to add it since ohNet needs to build from the command-line on both Windows and Posix, and can't depend on Windows users installing cygwin.

I think /usr/local/lib/ is probably more correct than /usr/local/bin/ for installing the shared objects.

However, I would caution that ohNet doesn't yet have a stable binary interface and hasn't yet established proper .so version numbering, so might not be wise to install to a system-wide location if there's any possibility of there being multiple applications installed that use different versions of ohNet. An alternative is to set the LD_LIBRARY_PATH environment variable when invoking your application. This environment variable can provide extra directories that are searched for shared libraries. I think this is generally frowned upon, in part because this environment variable will "leak" into any subprocess that you create, and also because it means that a sysadmin might have to worry about many copies of a library installed in different locations, making it hard to update them all when - for example - there's a security-critical fix. I think in this instance it's a reasonable approach to take, and indeed it's what is recommended by the Mono project for Mono applications that use native libraries: http://www.mono-project.com/Guidelines:A...Deployment

Also handy when figuring out why your shared libraries are not being loaded is to set LD_DEBUG=all. This will print out lots of information showing where the dynamic linker is looking when trying to find your libraries.

So, I would suggest either:

1. Put libohNet.so, libohNetJni.so and your app executable in "/usr/local/lib/YOURAPPNAME/", and put a shell script to start your application in "/usr/local/bin/" with something like this:

#!/bin/sh
LD_LIBRARY_PATH=/usr/local/lib/YOURAPPNAME/ /usr/local/lib/YOURAPPNAME/YOURAPPEXECUTABLE "$@"

2. Put libohNet.so and libohNetJni.so in "/usr/local/lib/" and put your application in "/usr/local/bin/". Be aware that you might have problems if you need to install on a system that might also have other applications installed that require their own version of libohNet.so in "/usr/local/lib/".
Visit this user's website Find all posts by this user
28-10-2011, 11:04 AM
Post: #3
RE: Shared objects on Linux
Hi Andrew,
Thanks very much for all the useful information and your suggestions. One correction to what I posted previously: my reference to /usr/local/bin was a typo, and I had actually been using /usr/local/lib as you suggested.

I've figured out how to get this working properly. The problem was that libohNet.so wasn't being built correctly. It should have an SONAME record, but it doesn't. I modified the link command that builds this in Common.mak to add the option -Wl,-soname,libohNet.so and I checked that this had worked by using the command objdump libohNet.so -p. If the libohNet.so file has been built correctly, the output from objdump will contain:

Dynamic Section:
NEEDED libpthread.so.0
NEEDED libstdc++.so.6
NEEDED libm.so.6
NEEDED libgcc_s.so.1
NEEDED libc.so.6
SONAME libohNet.so

The vital entry here is SONAME. With this in place, linking libohNetJni against libohNet.so produces a libohNetJni.so file that contains:

Dynamic Section:
NEEDED libpthread.so.0
NEEDED libohNet.so
NEEDED libjvm.so
NEEDED libstdc++.so.6
NEEDED libm.so.6
NEEDED libgcc_s.so.1
NEEDED libc.so.6

Now we have all the ingredients necessary to make this work. When the Java code calls
System.loadLibrary("ohNet");
System.loadLibrary("ohNetJni");
the linker finds the matching SONAME and NEEDED records in libohNet.so and libohNetJni.so and everything resolves correctly, with no need to put anything in /usr/local/lib or set LD_LIBRARY_PATH. Any name could have been used for SONAME and NEEDED, such as libohNet.so.1 to indicate version 1 of ohNet.

Simon
Find all posts by this user
31-10-2011, 12:09 PM
Post: #4
RE: Shared objects on Linux
I'm glad to hear that's working for you.

I think we will eventually include the SONAME records in our builds, but I'm not sure if that also requires making commitments to maintaining binary compatibility. We'll certainly reach that point, but I'm not sure if we're there just yet - Simon might be in a better position to make a call on this.

Just one question - if the shared object files aren't in /usr/local/lib or LD_LIBRARY_PATH, where are they? In the same path as the .class/.jar files? Is this a setup just for development or does it work for deployment too?

Lastly, I also see that there's a property "java.library.path" which plays a similar role to LD_LIBRARY_PATH but is specific to a Java process. Here's a mention of it:

http://www.chilkatsoft.com/p/p_499.asp

It sounds like you have the solution you need, but I thought this might be useful nonetheless.
Visit this user's website Find all posts by this user
31-10-2011, 02:08 PM
Post: #5
RE: Shared objects on Linux
(31-10-2011 12:09 PM)andreww Wrote:  I'm glad to hear that's working for you.

I think we will eventually include the SONAME records in our builds, but I'm not sure if that also requires making commitments to maintaining binary compatibility. We'll certainly reach that point, but I'm not sure if we're there just yet - Simon might be in a better position to make a call on this.

The word "eventually" doesn't sound very encouraging from my perspective. Sad Including SONAME doesn't imply a commitment to binary compatibility. It can be used to do that, if you add a specific version number to the SONAME value, but it can also be used without specifying any version information. I was using the flavour without version information, and the only effect of adding this is to make it possible for the references from libohNetJni.so to libohNet.so to be successfully resolved.

Quote:Just one question - if the shared object files aren't in /usr/local/lib or LD_LIBRARY_PATH, where are they? In the same path as the .class/.jar files? Is this a setup just for development or does it work for deployment too?

I am putting these in the same directory as my application jar files. I have written a custom Java classloader that redirects the System.loadLibrary() call to look in this location. This approach is suitable for both development and deployment, and I'm using it for both of these.

Quote:
Lastly, I also see that there's a property "java.library.path" which plays a similar role to LD_LIBRARY_PATH but is specific to a Java process. Here's a mention of it:

http://www.chilkatsoft.com/p/p_499.asp

It sounds like you have the solution you need, but I thought this might be useful nonetheless.

This doesn't solve the problem, because the failure occurs in the Linux linking step after both libohNet.so and libohNetJni.so have been found and loaded by Java. The problem is that the Linux linker can't resolve the references from libohNetJni.so to libohNet.so (even though libohNet.so has already been loaded by the same process) because of the missing SONAME/NEEDED records. Java has no control over this linking step, so setting java.library.path has no effect.

Simon
Find all posts by this user
31-10-2011, 03:56 PM
Post: #6
RE: Shared objects on Linux
(31-10-2011 02:08 PM)simoncn Wrote:  The word "eventually" doesn't sound very encouraging from my perspective. Sad Including SONAME doesn't imply a commitment to binary compatibility. It can be used to do that, if you add a specific version number to the SONAME value, but it can also be used without specifying any version information. I was using the flavour without version information, and the only effect of adding this is to make it possible for the references from libohNetJni.so to libohNet.so to be successfully resolved.
Ah, I have only limited knowledge of this mechanism, so it sounds like I might have been overly conservative here. I'll see if I can get Simon to comment on a timescale for incorporating the SONAME. If you have a patch and it definitely doesn't complicate version management I imagine we could get it in relatively quickly.

(31-10-2011 02:08 PM)simoncn Wrote:  I am putting these in the same directory as my application jar files. I have written a custom Java classloader that redirects the System.loadLibrary() call to look in this location. This approach is suitable for both development and deployment, and I'm using it for both of these.
I see. So once you've loaded libohNetJni.so explicitly from that directory, the dynamic linker will look in the same directory, discover libohNet.so and accept it because it has the correct SONAME in it. In contrast, if it doesn't have the SONAME set, it will continue to search in system directories, where it relaxes the SONAME requirement. Is that correct?

(31-10-2011 02:08 PM)simoncn Wrote:  This doesn't solve the problem, because the failure occurs in the Linux linking step after both libohNet.so and libohNetJni.so have been found and loaded by Java. The problem is that the Linux linker can't resolve the references from libohNetJni.so to libohNet.so (even though libohNet.so has already been loaded by the same process) because of the missing SONAME/NEEDED records. Java has no control over this linking step, so setting java.library.path has no effect.
Of course you're correct here. I knew there was a difference between Java loading the JNI library and then the JNI library loading the actual library, but I forgot about this distinction when reading that link. It's been less of a problem for me because the equivalent mechanism in Mono/.NET doesn't have two layers of native library.
Visit this user's website Find all posts by this user
31-10-2011, 07:22 PM
Post: #7
RE: Shared objects on Linux
(31-10-2011 03:56 PM)andreww Wrote:  Ah, I have only limited knowledge of this mechanism, so it sounds like I might have been overly conservative here. I'll see if I can get Simon to comment on a timescale for incorporating the SONAME. If you have a patch and it definitely doesn't complicate version management I imagine we could get it in relatively quickly.

Thanks, that sounds much more positive. Smile The patch is very small (about 20 characters added to one line of a make file) and I'll post it tomorrow.

Quote:I see. So once you've loaded libohNetJni.so explicitly from that directory, the dynamic linker will look in the same directory, discover libohNet.so and accept it because it has the correct SONAME in it. In contrast, if it doesn't have the SONAME set, it will continue to search in system directories, where it relaxes the SONAME requirement. Is that correct?

That's pretty close. I think the mechanism is that when libohNet.so is loaded into memory, the runtime linker adds its SONAME entries to an in-memory table and will then resolve matching NEEDED entries from subsequently loaded shared objects to the SONAME entries that it has already loaded, whether or not both libohNet.so and libohNetJni.so were loaded from the same directory. My evidence for this is that the same SONAME/NEEDED mechanism is resolving libohNetJni.so's reference to libjvm.so correctly, even though libjvm.so was loaded from a different directory. However, if there's no SONAME in libohNet.so, the NEEDED entry in libohNetJni.so is set at build time to an absolute path, which means that the runtime linker can only resolve it successfully if it finds libohNet.so at that exact location on the filesystem.

Simon
Find all posts by this user
01-11-2011, 08:31 AM
Post: #8
RE: Shared objects on Linux
Ah! I see - you pre-load the library into the process in Java and the SONAME is what ensures later requests for the library are fulfilled with the copy that's already loaded. Now it makes sense.
Visit this user's website Find all posts by this user
01-11-2011, 08:53 PM
Post: #9
RE: Shared objects on Linux
Here's the patch. It actually changes two lines, not one.

1) In Makefile, add a new macro definition:

linkopts_ohNet = -Wl,-soname,libohNet.so

I added this after line 107, on the assumption that it should be used for all flavours of Linux.

2) In Common.mak, line 386, change:

$(link_dll) $(linkoutput)$(objdir)$(dllprefix)ohNet.$(dllext) $(objects_core)

to:

$(link_dll) $(linkopts_ohNet) $(linkoutput)$(objdir)$(dllprefix)ohNet.$(dllext) $(objects_core)

I've tested this with Linux ARM on the latest GitHub version.

Simon
Find all posts by this user
10-11-2011, 04:21 PM
Post: #10
RE: Shared objects on Linux
Thanks for posting the changes. I've integrated them into our local copy so they'll appear on github pretty soon.

Note that I've only applied this change for Linux builds. Mac and other non-Linux Posix targets have an empty version of linkopts_ohNet for now. I also added an empty linkopts_ohNet to OhNet.mak to keep the Windows build working.
Find all posts by this user


Forum Jump: