libiconv Madness

| | Comments (4)

The other night, we got an AppleTV—a wonderful Christmas present from my parents and the replacement for all TV services in our house! No more Dish! No more Comcast!—and the first thing I did, of course, was install boxee. Though it works (it’s still pretty choppy when trying to stream Hulu, which is one of the things I want it to do most), I decided that if I had to stream from a shared drive somewhere, it’d be just fine.

Now, I run MacOS X Server in the basement, as some of you may know, and I decided that since it’s a server, it probably would be a good thing to share files with, ya know? Unfortunately, boxee only knows how to “speak” SMB and not Apple’s native AFP. (If I just lost you, you’re going to be really in over your head in a minute. But hang in there and maybe you’ll pick up something you can use to impress your spouse!) So I turned on SMB on the server. One click, right?

Um, no. I kept getting this error in /var/log/system.log:

[2008/12/03 22:56:29, 0] /SourceCache/samba/samba-187.4/samba/source/nmbd/nmbd.c:main(695)
  Netbios nameserver version 3.0.25b-apple started.
  Copyright Andrew Tridgell and the Samba Team 1992-2007
dyld: lazy symbol binding failed: Symbol not found: _iconv_open

Referenced from: /usr/sbin/nmbd Expected in: /usr/local/lib/libiconv.2.dylib

dyld: Symbol not found: _iconv_open
  Referenced from: /usr/sbin/nmbd
   Expected in: /usr/local/lib/libiconv.2.dylib

Argh. It kept failing with this every ten seconds, which is normal activity for a process that fails to start up. Unfortunately, whatever caused this error wasn’t going to go away on its own, so it was up to me.

I turned to my favorite troubleshooting tool, Google, and came up with jack-frickin’-diddly-squat. Nothing. Nada. Well, except for someone who was having similar problems, but for httpd, and the answer that that person got was only slightly short of useless.

So I did what any self-respecting geek would do: I dropped into the shell, dug out libiconv-1.12 (which yields libiconv-2.4.0… where does the madness end?!) and recompiled with the standard “configure/make/make install”. That changed nothing. Rats. Other services seemed to be unaffected, so I left things alone.

Well. This afternoon, while pounding a wire staple into the board that holds, among other things, the GFCI outlet that feeds the server, I tripped the GFCI, thereby restarting the server. Later on, for grins and giggles, I decided to look at the system log on that machine and saw…

Dec  4 13:29:11 shr-g5 org.apache.httpd[147]: dyld: Library not loaded: /usr/lib/libiconv.2.dylib
Dec  4 13:29:11 shr-g5 org.apache.httpd[147]:   Referenced from: /usr/sbin/httpd
Dec  4 13:29:11 shr-g5 org.apache.httpd[147]:   Reason: no suitable image found.  Did find:
Dec  4 13:29:11 shr-g5 org.apache.httpd[147]:   /usr/local/lib/libiconv.2.dylib: mach-o, but wrong architecture
Dec  4 13:29:12 shr-g5[1] (org.apache.httpd[147]): Exited abnormally: Trace/BPT trap
Dec  4 13:29:12 shr-g5[1] (org.apache.httpd): Throttling respawn: Will start in 10 seconds

Argh! The dreaded problem that the other guy had! I, however, had a secret weapon, no matter how flawed it may be, so without monologging/further delay, I engaged my secret weapon! Pew! Pew!

(It’s Apple’s Time Machine. Guess it’s not much of a secret.)

I just restored the libiconv that I had a few nights back, restarted the web service, and was relieved that it started normally.

So what was the difference between last week’s libiconv and this week’s libiconv? I’ll give you a hint:

shr-g5:apache2 admin$ file /usr/local/lib/libiconv.2.4.0.dylib 
/usr/local/lib/libiconv.2.4.0.dylib: Mach-O universal binary with 4 architectures
/usr/local/lib/libiconv.2.4.0.dylib (for architecture ppc7400): Mach-O dynamically linked shared library ppc
/usr/local/lib/libiconv.2.4.0.dylib (for architecture ppc64):   Mach-O 64-bit dynamically linked shared library ppc64
/usr/local/lib/libiconv.2.4.0.dylib (for architecture i386):    Mach-O dynamically linked shared library i386
/usr/local/lib/libiconv.2.4.0.dylib (for architecture x86_64):  Mach-O 64-bit dynamically linked shared library x86_64

And I’ll give you another hint:

shr-g5:apache2 admin$ file /usr/sbin/httpd
/usr/sbin/httpd: Mach-O universal binary with 4 architectures
/usr/sbin/httpd (for architecture ppc7400): Mach-O executable ppc
/usr/sbin/httpd (for architecture ppc64):   Mach-O 64-bit executable ppc64
/usr/sbin/httpd (for architecture i386):    Mach-O executable i386
/usr/sbin/httpd (for architecture x86_64):  Mach-O 64-bit executable x86_64

Note that when I was building the libiconv in a fury to get nmbd running that I did nothing more than “./configure” to kick things off. That’s when I remembered that the guy on the Apple discussions board who had a problem had noted that his libiconv only had one architecture in it… ding! If I remember correctly, they all have to match in numbers and types of architectures, even if you’re only going to use one of them.

Or at least, that’s what I think is going on.

So I dove back into my notes and did the following to build libiconv-1.12:

MACOSX_DEPLOYMENT_TARGET=10.5 CFLAGS="-arch ppc -arch ppc64 -arch i386 -arch x86_64 -g -Os -pipe -no-cpp-precomp" CCFLAGS="-arch ppc -arch ppc64 -arch i386 -arch x86_64 -g -Os -pipe" CXXFLAGS="-arch ppc -arch ppc64 -arch i386 -arch x86_64 -g -Os -pipe" LDFLAGS="-arch ppc -arch ppc64 -arch i386 -arch x86_64 -bind_at_load" ./configure --prefix=$PREFIX --enable-static

After doing a “make clean” (because things were dirty) and “make,” I checked to see what the sizes of last week’s libiconv and this week’s libiconv were.

And they matched.

So… deep breath… I did “sudo make install” and shut down the web server…

…and then restarted it…

…and it started without griping! I had solved the libiconv madness (at least as far as httpd is concerned).

I’m still nowhere with the nmbd problem and in spite of plenty of views on and, nobody’s been able to provide a clue.

In the meantime, I’ll just stream from one of my MacOS X machines which has SMB running—without a libiconv at all, strangely.

Updated 3-10-2009

OK, I have found the solution to the problem.

It turns out that the libiconv that I had in /usr/local/lib is not the libiconv that Apple supplies with MacOS X Server 10.5.x. That one resides in /usr/lib, and mine was completely missing. I don’t know where it went, nor how it disappeared.

The solution was to tarball the libiconv from a virgin OSXS 10.5.6 machine and plunk it down into /usr/lib, then delete (the usual, careful way—renaming all the files “.old” and then seeing how things work, then deleting them) the old ones.

I did get some help from Bruno Haible (one of the authors of libiconv) who pointed out that Apple had their own special version… but finding it on the Apple website was maddening. I had to use Google to find it. Anyway, if you want to build libiconv the way that Apple does, you’ll need to use the stuff on this page.

This builds a file whose structure is very different from the default configure/make/install that Bruno’s package provides. You can look at the output of nm on the resulting file to see that.

Anyway, problem solved. Hope this helps someone else.


Jay States said:

Can you apply Apple's patches to a newer version of libiconv? Exactly what do you do to install the apple version. Please email with instructions if you can. Big Thanks, J

Bill Eccles said:

Hi, Jay,

No, I haven't ever applied an Apple patch to libiconv (or anything else, for that matter). While I am pretty sure that I might be able to figure it out at some point if I need to, I haven't had the need arise... yet. I'd have to guess that you take each of the files in the Apple page and try to make them apply to the libiconv1.12 distro. But I'm not sure I'd want to have to do that, either...

However, the Apple version of libiconv1.11 produces libiconv.2.4.0.dylib, which is, by the definition of version numbers and libraries and whatnot, the same version number that the most recent distro of libiconv1.12 produces. By the rules, they should be functionally equivalent since the version number of the library didn't roll.

I'm running a weird system at the moment which works--and my update above lies a bit, as it turns out. As soon as I published the update, I realized I hadn't actually renamed and then deleted the old dylibs in /usr/local/lib, so I did that and the moment I did, libphp started to gripe that it couldn't find a symbol that it needed. I put them back. Sigh.

The solution was to leave both libiconvs in place, the Apple-provided one in /usr/lib, and the self-compiled version in /usr/local/lib. Apparently, in doing so, everybody remains happy. The binaries expecting "iconvopen" in /usr/lib find it there, and the binaries that are looking for libiconv in /usr/local/lib find what they need there, too. I'm presuming that any time I have something looking for libiconv, I'll have to point it at /usr/lib because the header files provided as part of XCode (I think that's the source) refer specifically to the symbols found in the Apple version.

It'll be interesting to see what happens when I have to recompile libphp the next time Apple does something to that. I don't know what libraries I'll have to specify in configuration. That could get interesting.


Matt Ryall said:

I had the same problem running the Ruby on Rails 'rails' executable due to two separate versions of libiconv, one provided by Apple and one in MacPorts.

$ rails foobar
dyld: lazy symbol binding failed: Symbol not found: _iconv_open
  Referenced from: /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/universal-darwin9.0/iconv.bundle
  Expected in: /opt/local/lib/libiconv.2.dylib

Thanks to your pointers, I managed to find a simple workaround: just change the DYLD_LIBRARY_PATH environment variable to point to the Apple one (/usr/lib) before running Rails.

$ export DYLD_LIBRARY_PATH=/usr/lib
$ rails foobar
      create  app/controllers
Bill Eccles said:

Great! Glad this helped somehow. And your response, now recorded forever in the Annals of Google, will certainly help other folks who have the same problem.

Thanks for posting!