libiconv Madness

The other night, we got an AppleTV—a wonderful Christmas present from my parents and the replacement for all TV services in our house! No more Dish! No more Comcast!—and the first thing I did, of course, was install boxee. Though it works (it’s still pretty choppy when trying to stream Hulu, which is one of the things I want it to do most), I decided that if I had to stream from a shared drive somewhere, it’d be just fine.

Now, I run MacOS X Server in the basement, as some of you may know, and I decided that since it’s a server, it probably would be a good thing to share files with, ya know? Unfortunately, boxee only knows how to “speak” SMB and not Apple’s native AFP. (If I just lost you, you’re going to be really in over your head in a minute. But hang in there and maybe you’ll pick up something you can use to impress your spouse!) So I turned on SMB on the server. One click, right?

Um, no. I kept getting this error in /var/log/system.log:

[2008/12/03 22:56:29, 0] /SourceCache/samba/samba-187.4/samba/source/nmbd/nmbd.c:main(695)
  Netbios nameserver version 3.0.25b-apple started.
  Copyright Andrew Tridgell and the Samba Team 1992-2007
dyld: lazy symbol binding failed: Symbol not found: _iconv_open

Referenced from: /usr/sbin/nmbd Expected in: /usr/local/lib/libiconv.2.dylib

dyld: Symbol not found: _iconv_open
  Referenced from: /usr/sbin/nmbd
   Expected in: /usr/local/lib/libiconv.2.dylib

Argh. It kept failing with this every ten seconds, which is normal activity for a process that fails to start up. Unfortunately, whatever caused this error wasn’t going to go away on its own, so it was up to me.

I turned to my favorite troubleshooting tool, Google, and came up with jack-frickin’-diddly-squat. Nothing. Nada. Well, except for someone who was having similar problems, but for httpd, and the answer that that person got was only slightly short of useless.

So I did what any self-respecting geek would do: I dropped into the shell, dug out libiconv-1.12 (which yields libiconv-2.4.0… where does the madness end?!) and recompiled with the standard “configure/make/make install”. That changed nothing. Rats. Other services seemed to be unaffected, so I left things alone.

Well. This afternoon, while pounding a wire staple into the board that holds, among other things, the GFCI outlet that feeds the server, I tripped the GFCI, thereby restarting the server. Later on, for grins and giggles, I decided to look at the system log on that machine and saw…

Dec  4 13:29:11 shr-g5 org.apache.httpd[147]: dyld: Library not loaded: /usr/lib/libiconv.2.dylib
Dec  4 13:29:11 shr-g5 org.apache.httpd[147]:   Referenced from: /usr/sbin/httpd
Dec  4 13:29:11 shr-g5 org.apache.httpd[147]:   Reason: no suitable image found.  Did find:
Dec  4 13:29:11 shr-g5 org.apache.httpd[147]:   /usr/local/lib/libiconv.2.dylib: mach-o, but wrong architecture
Dec  4 13:29:12 shr-g5 com.apple.launchd[1] (org.apache.httpd[147]): Exited abnormally: Trace/BPT trap
Dec  4 13:29:12 shr-g5 com.apple.launchd[1] (org.apache.httpd): Throttling respawn: Will start in 10 seconds

Argh! The dreaded problem that the other guy had! I, however, had a secret weapon, no matter how flawed it may be, so without monologging/further delay, I engaged my secret weapon! Pew! Pew!

(It’s Apple’s Time Machine. Guess it’s not much of a secret.)

I just restored the libiconv that I had a few nights back, restarted the web service, and was relieved that it started normally.

So what was the difference between last week’s libiconv and this week’s libiconv? I’ll give you a hint:

shr-g5:apache2 admin$ file /usr/local/lib/libiconv.2.4.0.dylib 
/usr/local/lib/libiconv.2.4.0.dylib: Mach-O universal binary with 4 architectures
/usr/local/lib/libiconv.2.4.0.dylib (for architecture ppc7400): Mach-O dynamically linked shared library ppc
/usr/local/lib/libiconv.2.4.0.dylib (for architecture ppc64):   Mach-O 64-bit dynamically linked shared library ppc64
/usr/local/lib/libiconv.2.4.0.dylib (for architecture i386):    Mach-O dynamically linked shared library i386
/usr/local/lib/libiconv.2.4.0.dylib (for architecture x86_64):  Mach-O 64-bit dynamically linked shared library x86_64

And I’ll give you another hint:

shr-g5:apache2 admin$ file /usr/sbin/httpd
/usr/sbin/httpd: Mach-O universal binary with 4 architectures
/usr/sbin/httpd (for architecture ppc7400): Mach-O executable ppc
/usr/sbin/httpd (for architecture ppc64):   Mach-O 64-bit executable ppc64
/usr/sbin/httpd (for architecture i386):    Mach-O executable i386
/usr/sbin/httpd (for architecture x86_64):  Mach-O 64-bit executable x86_64

Note that when I was building the libiconv in a fury to get nmbd running that I did nothing more than “./configure” to kick things off. That’s when I remembered that the guy on the Apple discussions board who had a problem had noted that his libiconv only had one architecture in it… ding! If I remember correctly, they all have to match in numbers and types of architectures, even if you’re only going to use one of them.

Or at least, that’s what I think is going on.

So I dove back into my notes and did the following to build libiconv-1.12:

MACOSX_DEPLOYMENT_TARGET=10.5 CFLAGS="-arch ppc -arch ppc64 -arch i386 -arch x86_64 -g -Os -pipe -no-cpp-precomp" CCFLAGS="-arch ppc -arch ppc64 -arch i386 -arch x86_64 -g -Os -pipe" CXXFLAGS="-arch ppc -arch ppc64 -arch i386 -arch x86_64 -g -Os -pipe" LDFLAGS="-arch ppc -arch ppc64 -arch i386 -arch x86_64 -bind_at_load" ./configure --prefix=$PREFIX --enable-static

After doing a “make clean” (because things were dirty) and “make,” I checked to see what the sizes of last week’s libiconv and this week’s libiconv were.

And they matched.

So… deep breath… I did “sudo make install” and shut down the web server…

…and then restarted it…

…and it started without griping! I had solved the libiconv madness (at least as far as httpd is concerned).

I’m still nowhere with the nmbd problem and in spite of plenty of views on support.apple.com and afp548.com, nobody’s been able to provide a clue.

In the meantime, I’ll just stream from one of my MacOS X machines which has SMB running—without a libiconv at all, strangely.

Updated 3-10-2009

OK, I have found the solution to the problem.

It turns out that the libiconv that I had in /usr/local/lib is not the libiconv that Apple supplies with MacOS X Server 10.5.x. That one resides in /usr/lib, and mine was completely missing. I don’t know where it went, nor how it disappeared.

The solution was to tarball the libiconv from a virgin OSXS 10.5.6 machine and plunk it down into /usr/lib, then delete (the usual, careful way—renaming all the files “.old” and then seeing how things work, then deleting them) the old ones.

I did get some help from Bruno Haible (one of the authors of libiconv) who pointed out that Apple had their own special version… but finding it on the Apple website was maddening. I had to use Google to find it. Anyway, if you want to build libiconv the way that Apple does, you’ll need to use the stuff on this page.

This builds a file whose structure is very different from the default configure/make/install that Bruno’s package provides. You can look at the output of nm on the resulting file to see that.

Anyway, problem solved. Hope this helps someone else.