All the hubbub surrounding the new iTunes store has eclipsed the fact that it’s now possible to share music collections via Rendezvous. I’m sitting at work listening to music stored on my Mac at home. All 16,000 tracks are available, with all the usual search functionality. All my playlists (both standard and “smart”) are available. I’m in hog heaven. If you set sharing on in iTunes 4 prefs, you can also provide a direct link into any point in your collection — Cmd-Click and select Copy URL. It’s also possible to link to specific items in the iTunes store. Check these examples.

iTunes 4 has arrived, and with it, the Music Store, AAC encoding and decoding, and artwork to go with your tunes.

Very cool. So how does it work? What follows is some stuff that might be handy for devloping helper apps, or just poking around.

Legal CYA note: I’m pretty sure this is all in line with the Music Store terms of service - this isn’t a derivative work since this just examines the very-public ID3 and QuickTime formats, and I’m not looking at ways to get around the security technology in iTunes 4. In fact, the Music Store doesn’t work for me (it’s never satisfied with my account information), so I don’t have any DRM’ed files to play with anyways.

.m4a file format

As an MPEG-4 offshoot, it shouldn’t surprise anyone that the AAC files now used by iTunes use the QuickTime file format, which was adopted for the MPEG-4 file format. In an OnJava article a few months ago, I examined this format and provided an all-java parser for the format.

Here’s what a typical .m4a file looks like when run through the parser:

ftyp (32 bytes)
moov (73049 bytes) - 3 children
  mvhd (108 bytes)
  trak (54805 bytes) - 2 children
    tkhd (92 bytes)
    mdia (54705 bytes) - 3 children
      mdhd (32 bytes)
      hdlr (34 bytes) [/soun - ]
      minf (54631 bytes) - 3 children
        smhd (16 bytes)
        dinf (36 bytes) - 1 child
          dref (28 bytes)
        stbl (54571 bytes) - 5 children
          stsd (103 bytes)
          stts (24 bytes)
          stsc (40 bytes)
          stsz (51908 bytes)
          stco (2488 bytes)
  udta (18128 bytes) - 1 child
    meta (18120 bytes)
mdat (4819123 bytes)

The big picture is that there’s a moov (a “moo-vee”, get it?) with a single audio track. The structure here, with the stbl sample table that describes where to get chunks of from the raw mdat media data is pretty typical.

But notice that udta atom. That’s “user data”, a wild and wooly place to hide anything you darn well like. My parser doesn’t think that meta is a container and the parsing stops there… meta isn’t described in the QT file format doc, so maybe it’s an MPEG-4 atom or maybe just a proprietary Apple format.

At any rate, the contents are clearly our music metadata Here’s some character data from the right column of a hexdump -C:

.>.ilst...;.nam.
..3data........A
eris' Theme (Orc
hestrated Versio
n)...%.ART....da
ta........Nobuo
Uematsu...8.alb.
..0data........F
inal Fantasy VII
 Reunion Tracks.
...gnre....data.
............ trk
n....data.......
.............cpi
l....data.......
......tmpo....da
ta.............2
.too...*data....
....iTunes v4.0,
 QuickTime 6.2..
..----....mean..
..com.apple.iTun
es....name....iT

The use of 4 bytes of size, and 4-byte names like genr make this structure look like normal Quicktime atoms (the nam and alb are preceded by the same byte, 0xA9, so these are probably 4-byte name constants too), so maybe this is some MPEG-4 thing I don’t know about (yes, I’m too cheap to buy the real MPEG-4 docs). At any rate, it looks eminently parsable.

Cover art

iTunes still uses version 2.2 of the ID3 MP3 metadata standard, as indicated by the first four bytes of a tagged file: 49 44 33 02, or ID3 followed by the version byte 0x02. According to the standard, a picture can live in a PIC frame, and if you look for it with a hex editor, that’s exactly what you’ll find. This will be followed by 8 bytes (three for frame size, one for text encoding, three for image format [iTunes seems to always be JPG], one for picture type [probably 0x00 for “other]), then a null-terminated description string (looks like iTunes doesn’t use the String, so there’s just a 0x00), and then the image data. Plenty easy to find or write once you look at the standard.

Image formats

Interesting thing I’ve noticed. Take a .gif file. Look at its header. The first six bytes should be GIF89a. Now use this file as your album art for a track in iTunes. Something interesting has happened.

I took Roxy Music’s notorious and outrageous cover to Country Life, turned the JPEG into a GIF with GraphicConverter, and dropped it on to the 10 tracks of the Country Life album. Then I dug in to one of the MP3’s and found the PIC frame. After the header, GIF89a is not there. Instead, it’s 0x89 followed by PNG. Looking further, we see the chunks IHDR, PLTE, IDAT… yep, it looks like PNG format to me. I found the same thing had happened in the presumable covr atom of the AAC file - a dropped GIF has become a PNG.

Which is interesting because this suggests iTunes has quietly performed a GIF to PNG conversion when I dropped in the artwork. Conspicuously or coincidentally, Jaguar’s “Preview” app includes JPEG and PNG among the formats it can convert to, but does not export to GIF. Perhaps Apple has quietly decided to burn all GIFs. If so, my hat’s off to them.

Music selection

One non-technical note you’ll have to oblige me, as I bust out the “when I was your age” cliches…

Some are complaining about the music selection in the iTunes Music Store, but frankly, I’m pretty impressed. When compact discs came out in the early 80’s, there were virtually no pop or rock CD’s available, and some people said the format was too good for anything but classical and maybe jazz. Pop CD’s eventually did emerge, but some were a long time coming. In particular, the Beatles didn’t hit CD until about 1987, by which time CD players were rather common in the US. Today, the Beatles aren’t in the Music Store, but give it some time. Frankly, I’m more worried about the small-label artists, both the young punk bands on hole-in-the-wall labels, and the classic artists like Todd Rundgren, Elvis Costello, and Frank Zappa, whose stuff is on specialty reissue labels like Rhino and Rykodisc. Hopefully the Music Store will eventually stock their stuff too.

Any other neat stuff in iTunes files you’ve found? Any good ideas for ways to add value to your music collection?

Editor’s Note:
These are very rough, bullet-list style notes…please excuse any typos.

Strange Noises Coming From Cupertino

Well, it’s Monday, April 28th. The day of Apple’s big announcement regarding it’s rumored music service. Not being in California for announcements like this was lessened a bit by the live simulcast available at the Apple Store, located in the CambridgeSide Mall in Cambridge, MA.

Some highlights from the announcement are as follows:

• Songs played…Soak Up the Sun (Sheryl Crow), and “The Times Are Changin’” (Bob Dylan)
• Jobs quotes from Mossburg’s review of the 17-inch PB
• Over 3 million downloads of Safari
• Mentions iLife and Keynote
• Announcements of FCP4

Innovate in Music

• Replayed the advertisement called, “Rip, Mix, Burn” that aired in 2001
• RMB ad was more of an announcement for iTunes, which then lead to the the release of the iPod
• Apple received a Grammy in 2002, which was the first ever awarded to a computer company
• Some people thought “Rip” meant “rip off”, but it means to take the data from a CD and place the data (bits) on your machine.
• Acquire, Manage and Listen are the new terms, replacing RMB
• Sold 700K iPods since their release, and is the #1 MP3 player in the world.
• In the digital age, the iPod is doing the same thing that the Sony Walkman did…

New iPods

• Three new sizes 10/15/30 GB (30 holds 7500 songs)
• New Slim line (0.62 inches) 5.6 ounces (less than the weight of 2 CDs in their jewel cases), and thinner too
• All of the controls are touch-based.
• Better display, buttons moved above the scroll wheel
• Firewire connector is on the bottom, and it also has a Dock so it can connect to the computer (Sync, Charge, and Go)
• The Dock has a line-out so you can hook it up to your stereo
• Lighted buttons on the iPod
• USB2 is added for Windows customers
• Software for Windows still needs some testing
• New software update in June
• New cable for connecting. With USB, there’s a dual-plug so it connects to your computer, and into the charger ($19 extra for the USB2 cable)
• Mac and Windows iPods will ship in the same box
• Auto-Sync (patents pending)
• New Features:
• AAC decoding (invented by Dolby)
• On-the-Go playlist (build playlists on the iPod)
• Personalize the Main Menu
• Alarm clock (beep or play playlist)
• New games (Solitaire and Airship)

• Jobs then demo’d the new iPod…
• On-The-Go playlist demo…
• Select song
• Hold button down
• Title flashes, indicating that the song/playlist has been added to the OTG playlist
• Settings –> Main Menu (all of the key things have on/off toggles so you can customize the main menu)

New iPods are thinner and lighter, include a dock, USB2 (if needed), auto-sync, and OTG playlists

The most popular iPod has been the 10GB iPod because of its price

• 10 GB: 2500 songs ($299)
• 15 GB: 3700 songs ($399)
• 30 GB: 7500 songs ($499)

New iPods will be available this Friday, May 2 at Apple Stores throughout the US.

Manage

• iTunes 4 is released…
• Adding AAC encoding (standard of the MPEG-4 encoding)
• MP3
• Adding support for Rendezvous
• iTunes will use Rendezvous so you can add access to playlists on other Macs and play them on your Mac
• DVD archiving so you can backup your music library
• Album Art for album/CD covers

Acquire

• We acquire our music from CDs
• Napster came along and was shut down in 2001
• Near-instant gratification…but taking music from online sources is “stealing”
• No legal alternative (but there is pressplay and Rhapsody, but they have thier own limitations)
• Subscriptions are the wrong path…
• People have bought their music for as long as can be remembered…
• People want to buy downloads, just as they have in the past…
• Apple created the “Music Store”

• Quoted Hunter S. Thompson about the music business…
• 200,000 tracks are currently available in the Music Store, and will continue to load them
• Rights for users:
• Unlimited CD burns for personal use
• A “governor” has been placed, restricting you from burning a playlist more than 10 times without modifying that playlist
• Can put the music on an unlimited number of iPods
• Play on up to three Macs
• Authorization can be transferred to new systems
• Use music in any of the applications
• Purchase for $0.99/song with no subscription fee

• Songs are encoded in 128 Kbps
• Free 30-second previews in 128 Kbps
• Album cover art is included with the downloads

Jobs: “It’s not stealing any more, it’s good karma.”

Beyond…

• Online Apple Store does $1.2B in sales/year
• Songs can be purchased with one-click via the online Apple Store
• Browse mode in iTunes
• Exclusive tracks that no other source has
• iTunes Music Store is built into iTunes 4
• Steve demos iTunes 4 and the iTunes Music Store…
• One of the coolest features of the new version of iTunes is the ability to view music videos right within iTunes’ interface as a QuickTime stream.
• Played videos from Sum 41 and Bob Dylan

And the clincher….(dum-dum-dum….)

• Only Apple could have created this online Music Store…
• Initially only available in the US…available starting today…requires a credit card with a US billing address (there’s your loop around if you’re overseas)
• iTunes 4 is a free download from apple.com
• Free software upgrade for your iPod is available today…

iTunes and the Music Store will be available for Windows users by the end of the year…

What do you think of Apple’s new music service?

The news we get is influenced by the people and organizations that bring it to us. One of the great things about blogs is that we can pick our news sources at a more granular level. When I configure NetNewsWire, I’m building my daily newspaper. These are the people who tell me about what’s happening.

What if you could choose the people who covered the White House, Technology News, or your favorite Sports Team? I remember growing up on David Brinkley and Chet Huntly and the NBC teams covering political conventions, space shots, and everyday life. My grandparents depended on Cronkite and CBS. We couldn’t pick and choose reporters on an individual basis so we had to settle on the network their assembled staff (Ellerby, Utrick,…) as a whole.

It’s a little different now in these days of remote controls. We can flip back and forth to catch a story or reporter we’re watching for on another network. When I was a kid, I was the family remote control (Daniel, flip over to channel 3). Fortunately, my parents were pretty locked in to one network. It was a big deal when they switched over to PBS and McNeil/Lehrer.

I don’t watch much TV news. I just don’t like it very much. Until recently I got all of my national news by listening each day to NPR’s Morning Edition and All Things Considered. Now I subscribe to them on Audible and flip past stories I don’t care about.

Blogging and news aggregators allow me to assemble my own daily newspaper with my favorite reporters and commentators. I have them divided up into categories that look much like the sections in a newspaper: World News, Technology, … and I have a category called “The Guys” that would be on my own blogroll if I had one.

Let’s organize our own convention coverage. The next conference I’m attending is JavaOne. Suppose you could assemble a team of bloggers to cover JavaOne. Who
would you choose?

Who would be on your JavaOne coverage dream team?

Come Together….

Convergence. Remember when the big buzz word was convergence? Of course, back then it meant that your digital toolbox would gradually shrink to just a set of Swiss Army knife-like devices. Yes, your toaster would finally be able to communicate with your coffee maker and your alarm clock to prepare your morning breakfast. Well, despite the usual IT industry fluff, the real definition merely involves the concept of multiple things coming together.

Mac rumor sites work the same way. At first, it’s a series of unrefined whispers and gossip. Slowly, these hints and gossip take form and solidify into genuine rumors. The result is that the rumor sites converge on an agreed set of hearsay. If you haven’t been paying attention (which would mean that you weren’t listening to me), the latest buzz is about an Apple music service.

Loose Lips Sink Ships

So at first we just had whispers of a music service. Then the details started to fill in. Major players started to say too much. I imagine the Mercurial One took anger at this, but what can you do? A deal this big would have to involve a good number of people.

In case you’re still in the dark, Apple is rumored… okay, let’s face it, Apple is /going/ to be debuting an online music service. The “facts” thus far point to a service which is of high-quality design and simplicity of use. The service is going to have music from the five major recording labels. Apple’s been in talks with the various record execs and has secured the necessary rights to distribute the music.

Additional details involve the use of AAC for the audio format and tight integration with the Apple Store’s 1-Click (R) sales system. If you’ve never used it, take a look at iPhoto. iPhoto’s print and book ordering system uses 1-Click to simplify payment. Your credit card information is stored on Apple’s systems (both pros and cons from a security standpoint) and gets billed for each transaction.

iCame for the iMusic, iStayed for the iPod

Alongside the release of the music service (and a requisite iTunes update to support the service), rumors point to a new iPod. The rumor site Think Secret reports that these will be 15GB and 30GB units with a new docking station. No thoughts on FireWire 800, but apparently USB 2.0 is going to be thrown in there somehow.

I imagine that to play AAC, an iPod will need a firmware update at the very least. Hopefully Apple won’t alienate its current iPod owners by requiring a new unit to work with the service. Something like that would cripple the debut of the service. They need to build on the success of the iPod, not try to force it to happen all over again.

Music to Our Ears

The big event is said to be this coming Monday, the 28th. Hopefully there will be 10-20 free downloads to whet our appetites. Beyond that, it seems the downloads will be $1 a piece, with little to no control of the resulting file to be imposed upon the consumer. I’m crossing my fingers for that MWNY2002 Rendezvous enabled iTunes, too. Looking forward…

Are we on the verge of Apple reinventing the music industry?

Here are my notes from Zeev Suraski’s keynotes (corrections href="mailto:phpcon2003@jepstone.net">are welcome):

How did we get here?

From Zeev’s perspective, it began with a
little shopping cart. He and his colleague needed a quick & dirty
shopping cart app for a University research project.

So, they used PHP/FI 2.0. They found that it was inconsistent and
error prone (syntax errors turned out to be PHP problems, not
problems in their code). So, they jumped in and helped out, and PHP
3 was the result (a flexible, consistent, and extensible scripting
language).

PHP3 and PHP4

PHP3 was revolutionary, had surprising growth, but disappointing to
Zeev and his fellow developers (they felt it wasn’t up to the tasks
that other people put it to). Performance was a concern (for
example, it interpreted as it executed, so if you called a function
1000 times, it had to be interpreted 1000 times).

So, they developed PHP4 and the Zend engine (modular, fast–compiled
to an in-memory representation first, reference counting, improved
extension APIs, and thread-safety).

OO support had some problems in PHP4. Objects defaulted to
pass-by-value (”evil clones” of the objects). So, you needed to use
explicit pass-by-reference. WTMA problem: Way Too Many Ampersands.

PHP5

PHP5 (Zend Engine 2) uses a handle-based object model (easy to move
objects around, better performance, better memory usage, supports
more OO features). They plan to offer a compatibility mode with
older scripts.

New OO Features: visibility levels, abstract methods/classes,
interfaces, type hints (adding some bits of typing to PHP, but just
for objects), namespaces/import (maybe), exceptions, static members,
unified constructors, destructors, instanceof operator, per-class
constants, autoload (lazy class loading), overloading/accessors,
better integration with Java, .NET, GTK+

Other stuff:

• Beta soon
• Advanced Data Types (hashtables, trees, etc)
• Remote Procedure Calls (Java, .NET, Corba, etc)
• XML improvements
• mysqli
• PHP-GTK 2

Links:

• engine2@lists.zend.com
• php5-dev@lists.php.net
• internals@lists.php.net
• https://www.zend.com/phorum/
• https://www.zend.com/zend/future.php

Here are my raw notes on Rasmus’ keynote talk from PHPCon East 2003 (corrections, additions, etc. welcome):

PHP History

10 Years ago, Rasmus wrote CGI programs in C. Lots of code to solve
a simple web problem. He says you couldn’t expect your HTML
designers to go in, edit your C code, and recompile. So, lots of
folks switched to Perl and CGI.pm.

Still, he argues that Perl and CGI wasn’t something that an HTML
designer could jump in and start editing.

What he wanted was PHP; code embedded in the HTML: “A block of raw
HTML followed by the minimum amount of logic possible.”

In PHP’s early days, he’d read the html file on disk using seeks,
and this had a huge penalty in things like loops, where PHP would
have to go back and forth in the file. Then he discovered mmap(2).

He figures that since he didn’t know how to write a scripting
language, PHP couldn’t have been a scripting language :-) (until PHP3,
when people who knew how to write scripting languages got involved).

PHP Adoption

He feels that PHP caught on because it solved 80% of the web problem
without requiring you to master a programming language; over
time, that missing 20% got added.

PHP has grown; he says that 3 out of 10 sites are PHP-driven. That
is based on Netcraft, so I wonder if that means that they are
PHP-enabled. Is PHP enabled by default on most or many Linux
distributions? All the Unix hosting providers I’ve seen support PHP, but being PHP-enabled doesn’t mean a site is necessarily using it. But I think it’s still pretty widely-used (I use a mix of PHP, Perl, and other random stuff on www.jepstone.net).

A Good Solution

He says that a good solution should have a shallow learning curve,
instant gratification, build on what you know and have great
documentation. It should solve simple problems easily, but also be able
to solve the most complex of problems.

It should be secure, borrow from the best, and run everywhere.
(being free is great, too).

He also says that a good solution should teach the basics by not
hiding the problem (he says many GUI-based tools obscure the
problem–he’s bashing Cold Fusion for letting the user point and click for the simple stuff and then requiring the user to
ramp up with their markup language when it’s time to do something
more complex).

Application Architecture

He’s pushing an architecure like this (a discipline for PHP
programmers):

• HTML Templates ($DOCUMENT_ROOT/*.php) (5% PHP, 95% HTML)
• Template Helpers ($DOCUMENT_ROOT/*.inc) (header functions, footer functions, common layout elements)
• Business Logic (/usr/local/php/*.inc)
• C/C++ Core Code (stuff you couldn’t do in PHP; most apps won’t
  have this)

Related link: https://birdhouse.org/blog/archives/000810.php

What the… ?

Why has iSync got its fingers in Safari’s bookmarks? Don’t see any evidence of them on my iDisk, iPod, or in iCal. Theories?

Related link: https://jicyshout.sourceforge.net/oreilly-article/

According to ls -l, tomorrow is the first anniversary of some code I wrote that eventually led to jicyshout, an open-source library to enable the Java Media Framework to play http-based MP3 streams.

By “http-based MP3 streams”, I’m talking about Shoutcast and Live 365, mostly, both of which use a similar protocol for streaming web radio. Stations using these systems aren’t big commercial stations, which tend to use DRM’ed Real or Windows Media to play the same 50 crappy songs you never need to hear again (especially “Hotel California” and “Gold Dust Woman”). Instead, they tend to be tiny niche-casters broadcasting real jazz, anime and j-pop, an all-Zappa playlist, etc. In other words, with the exception of KFOG’s “10@10″, Shoutcast and Live 365 stations are the only webcasters I can stand to listen to.

So I started experimenting with the JMF to figure out why it could play MP3 files, locally or off a server, but not streams. JMF has rtp/rtsp streaming support, but that’s not what we’re talking about here. The http streams are easier for a client, because you never have to worry about dropped data - you’re guaranteed to get everything. In this standard, the heavy lifing is on the server side.

What I found was that the class that plays MP3s is com.sun.media.content.unknown.Handler, so I set out to get him to be able to play the http stream. What I eventually figured out is that JMF is very dependent on the protocol and filename extension of a URL to decide what Player, if any, can handle the content, and it couldn’t figure out what to do with something like https://147.4.220.89:8000 (Fanelia.com RPG Radio) or https://www.live365.com/play/45939 (Anime Hardcore 1).

Oh and during this time, my son was born with a totally scary heart condition that required surgery four days after birth, so I put this work down for the three weeks we were at the hospital. Shortly after that, I applied to be a presenter at the O’Reilly Mac OS X Conference and amazingly (since my MP3 player wasn’t done yet), they accepted my talk on java media.

Long story short - I got it to work in a somewhat nice JMF kind of way, and while I was reading the stream myself, I threw in parsers for popular metadata formats like ID3 and Shoutcast.

The conference wanted speakers to write a companion article for one of the O’Reilly websites, so I whipped up a pair of articles using jicyshout as the sample code - the first was about the JMF side of how it works, the second was about the various metadata formats.

Then the stupid thing happened.

There was apparently some sort of licensing fiasco regarding MP3’s, and Sun responded by making JMF unavailable for a few weeks, then posting a new 2.1.1b version that removed MP3 support.

Obviously, this had a fairly lethal effect on both jicyshout and my articles about it. Nobody who downloaded the code could play MP3’s with it if they didn’t already have a pre-2.1.1b JMF, making the code and the articles instantly worthless. I decided to hold the articles for a while to see if Sun ever put the MP3 support back - it was, after all, practically the only useful format suppored by JMF - but it never happened. Indeed, from the looks of the website and the lack of new content in the last seven months, JMF seems almost utterly abandoned.

So, now that it’s been a year, I’ve posted the articles to the jicyshout website in hopes that they’ll do someone some good. They’re unedited and somewhat dated - ignore everything about icecast, which has dropped MP3 in favor of Ogg Vorbis in the past year. But on the other hand, the shoutcast in-stream metadata format is very poorly documented, and I have a decent write up of that (and code to parse it!), so the more we can disseminate that information, the better.

Web radio in java… what was I thinking? Does this project do anyone any good at all?

The O’Reilly Emerging Technology Conference runs from Tuesday, April 22 through Friday, April 25 at the Westin Santa Clara. The best seat in the house is the front row of the Santa Clara Ballroom. But if you can’t make the show, then here’s the hitchhiker’s guide to all of the breaking news.

• The Conference News page features daily photos (by yours truly), weblogs, and news stories. If you want to keep up with events as they happen, this should be your first stop.
• The Sessions and Events grid provides an overview of the entire week, including room assignments and start times.
• The Conference Home Page provides an overview of the show and includes complete registration information.
• The Exhibitors List is a real who’s who of who’s cool. You can register for a Free exhibit hall pass if you want to stop by and check things out. The Exhibit Hall is open on Wednesday and Thursday from 10 am to 4 pm.
• For related O’Reilly content, go to OpenP2P.com. This site is the ongoing hub for O’Reilly’s emerging technology content.
• The Birds of a Feather schedule is filling out nicely, and you might want to take a look.
• For media inquiries, contact Suzanne Axtell.
• Go to the Mac OS X Innovators Contest page for more information about the award presentation on Thursday, and how to enter the next contest that begins on May 8, 2003.

If you’re in the Silicon Valley area, I urge you to stop by for at least the exhibits (Wednesday and Thursday), and ideally for the sessions and keynotes (Wednesday through Friday). This is truly an exciting event.

If you have questions or comments about the Emerging Technology Conference, please post them here.

OSNews recently had two interesting interviews with the leads of the two major Mac OS X Unix porting projects. First up was an interview with Max Horn, who leads the Fink project. The second was an interview with Jordan Hubbard, who leads the DarwinPorts project (and also works at Apple). Both interviews discuss the various goals of each project and where they want to go in the future.

I’ve been a long time Fink user, but after reading a bit about DarwinPorts I decided to give it a try. Honestly I don’t have much loyalty to either project, I just want to get my Unix software working easily. I think this helps me have a relatively unbiased view. Below are descriptions of using the two different tools for the most common tasks: Installation of the tool, package installation, package removal, package upgrading, and searching for packages. If you’ve never used either of these tools, there’s just about enough information here for you to get started.

DarwinPorts

DarwinPorts is based upon the FreeBSD ports system. However, while FreeBSD’s ports make use of facilities of the make utility, DarwinPorts has been written from the ground up to use the Tcl language. According to the DarwinPorts FAQ, this allows for some interesting information to be extracted from the packages.

DarwinPorts automatically downloads the source files for your chosen package, as well as the source files of packages it depends on. These source files are then compiled locally and installed directly into a quarantined area of your file tree. The DarwinPorts project has also created many Apple Installer packages of their packaged applications. These are available for download and installation from packages.opendarwin.org.

DarwinPorts: Initial Installation

To install DarwinPorts, you must make use of the Concurrent Versioning System (CVS). These Terminal
commands should get you started:

% cd ~/Documents
% cvs -d :pserver:anonymous@anoncvs.opendarwin.org:/Volumes/src/cvs/od login
% cvs -d :pserver:anonymous@anoncvs.opendarwin.org:/Volumes/src/cvs/od co -P darwinports

There is no password for the anonymous account, so when the first cvs command asks you for a
password, just press Enter. The second command will download the darwinports project from the CVS
server. This process may take a few minutes, depending on your connection speed and how busy the
server is. Once the check out has finished, the following commands will finish up the installation:

% cd darwinports
% make
% sudo make install
% sudo pico /etc/ports/sources.conf

You should now be presented with the contents of the /etc/ports/sources.conf file within
the pico editor. Move the cursor to the end of the line and create an entry that is similar to
file:///Users/username/Documents/darwinports/dports, replacing username
with your user account’s short name. Press the Ctrl+O keys, the Enter key, and then the Ctrl+X
keys. You should now have a base install of DarwinPorts in the /opt/local directory. Do not move
the ~/Documents/darwinports folder around until you are more familiar with the various
DarwinPorts tools.

DarwinPorts: Installing Packages

The two Unix applications that I use most often are xchat and pan. DarwinPorts contains two versions
of xchat: 1.8.11 and 2.0.1. I’ll be using the latter, which has a package name of xchat2. There is
only one version of pan, which is version 0.13.4. To install these packages, use the port
command as follows:

% sudo port install xchat2
% sudo port install pan

DarwinPorts will check for any dependent libraries and applications of each app and install them as well, all automatically. Once a port is installed, issue the command sudo port clean package to reclaim disk space used during the build process. For example, after installing xchat2, I’d enter the command sudo port clean xchat2 to clean up the build files.

DarwinPorts: Removing Packages

Removing installed packages is a piece of cake. Just type in the command sudo port
uninstall package. To remove my install of xchat2, I’d use the command
sudo port uninstall xchat2. Easy enough, eh?

DarwinPorts: Upgrading Packages

To upgrade a package, you must first remove the existing package as per the previous section. Once
that has been completed, you’ll want to issue the commands below. These commands use cvs to update
your dports collection to the latest releases. Once it’s completed, you simply use the install
procedure above to install the upgraded package.

% sudo port uninstall xchat2
% cd ~/Documents
% cvs -d :pserver:anonymous@anoncvs.opendarwin.org:/Volumes/src/cvs/od up
% sudo port install xchat2

DarwinPorts: Finding Software

If you want to see what software packages are available through DarwinPorts, use the following
command port search .+ | more. You will be provided with a list of the
packages, their available versions, and a brief description of each. You can also search for
a specific package by supplying all or part of its name: port search xchat
will reveal the two available xchat packages.

Fink

Fink is based upon Debian GNU/Linux’s package management tools: apt and dpkg. The apt tool is used to retrieve packages and their dependencies from the Internet while dpkg does the actual work of installing the various files. Fink takes this a step further by adding the ability to create dpkg packages from source tarballs, compiling the software right on your computer.

Fink: Initial Installation

There are two methods for installing the Fink package. The first uses a binary
installer, which is the easier of the two methods and is recommended for users
who are not as comfortable with the command line interface. Simply download the disk image from the Fink website, mount it with Disk Copy, and then run the Installer package on the image. Follow the prompts and you’re on your way to open source goodness.

The second method uses a source distribution. The Fink website has detailed
instructions for installing the source distribution. Both versions have the same tools, features, and packages, so pick a distribution that you are comfortable with.

Fink: Installing Packages

There are also two methods of installing software packages with Fink. I recommend picking a single method and using it consistently. If you want to save time and don’t mind running slightly older releases, use precompiled packages. If you spending a little more time to install the latest releases of software packages is okay with you, use the source packaging. It’s all a matter of personal preference.

You can install precompiled binaries by using the apt-get tool. To do this, use the command sudo apt-get install package. For example, to install xchat, I’d use the command sudo apt-get install xchat. The apt-get tool will go out onto the Internet and download the binary package and then pass it to dpkg for installation. It will also download any dependent libraries or applications and install them as well. One advantage of Fink’s package installation is that you can specify multiple packages on the command line, such as sudo apt-get install xchat pan.

Fink’s second method of installing software installs from application source. Fink will download the source tarball for the application and compile it. It will then package it and pass the resulting package to dpkg for installation. In case you’re wondering, one reason to install a package the second way is that the version of the package in the source method is often newer than the version available as a precompiled binary.

To install a package via the second method, enter the command fink install xchat. Just like the apt-get method above, all dependencies are met and installed. (A quick side note: If you are using Apple’s X11 package, you want to install the system-xfree86 package before installing any X11 programs through Fink). Also like the apt-get method, you can specify multiple packages in a single command.

Fink: Removing Packages

Removing a package through Fink is easy. Use sudo apt-get remove package or fink remove package. Both commands can be used interchangeably to remove packages installed by either method. Multiple packages on a single command line are possible here as well.

Fink: Upgrading Packages

In order to upgrade your packages in Fink, you must first update your package listing. With the precompiled packages, this means using the command sudo apt-get update. For source packages, use the command fink selfupdate-cvs. Once package listings are updated, use sudo apt-get upgrade to upgrade your precompiled software or fink update-all to upgrade your source compiled software. It’s worth noting here that these commands will upgrade all software packages in their control. You can update individual packages by specifying which to upgrade. A man on fink or apt-get will get you the right information.

Fink: Finding Software

If you’d like a listing of software packages available through Fink, use the command fink list | more. Much like DarwinPorts, you’ll see a list of packages, versions, and descriptions. You can also look for specific packages by specifying a package name following list (and dropping the | more).

The Verdict

While it is truly a matter of personal preference, I feel that the Fink project offers an easier install, easier package management, and a larger selection of packages. The fact that I can easily upgrade all of the Unix packages in my Fink distribution with two commands seals the deal. With DarwinPorts, I’d need to issue two commands per package (uninstall, then install, plus a clean) in addition to sychronizing my cvs repository.

That being said, DarwinPorts is perhaps “truer” to Mac OS X’s heritage. Darwin is technically a BSD and Ports is about as BSD as you can get. Well, without counting /etc/rc files ;) Also, apparently DarwinPorts runs on other BSD distributions, quite an interesting venture!

I’m Not Satisfied

I’ve encountered both apt and ports in their original environments (Debian and FreeBSD, respectively) in the past. One problem I found with Debian is that their packages tend to run a little bit behind (i.e. older). I think that Fink has circumvented this issue by adding the ability to compile from source directly. FreeBSD’s ports is a wonderful system, but I feel that it requires a bit more effort than should be needed to keep your software up to date.

After fiddling around with those two OS’s, a friend turned me on to Gentoo Linux. Gentoo uses a package management system called Portage, which is also inspired by the Ports system. Portage handles software installation similar to how Fink does. It keeps a repository of available packages which are synchronized via rsync (instead of CVS) and then builds packages from source. It can also upgrade multiple packages with two commands (emerge sync && emerge -u world). I find this to be quite convenient.

The other thing I like about Gentoo is that it keeps a system-wide make.conf file. This file allows you to enter gcc optimization flags as well as USE flags. Specifying gcc flags allows you to build for your specific processor architecture. For example, you can specify that you have a G4 processor and want to make use of its AltiVec unit. Upon compiling a package, these extra flags will be specified, giving you that extra bit of optimization. Conversely, DarwinPorts and Fink both build for G3 processors for compatibility’s sake

Gentoo’s USE flags allow you to specify what kind of features you want to compile into your application. As an example, you might specify USE=”-X -gtk -gnome -kde -ipv6 tcpd”. Upon compiling the application, support for X11, GTK, GNOME, KDE, and IPv6 would be left out, but support for tcp wrappers would be put in. This allows you to drop unwanted features for a small performance gain and resource decrease.

Closing Up

With these tools in hand, I challenge you to go out and discover the world of Unix applications. There are a lot of tried, tested, and true tools out there just waiting to gain more users. With tools like Fink and DarwinPorts, installing these Unix utilities is easier than ever (especially when you look at the GUI management tools that are being developed for both projects). While you’re at it, see if you can get someone to start a project which works on porting Portage to Mac OS X :) Enjoy!

P.S.

This is my second time writing this entry. I actually had my first Safari crash since the v73 beta was released. “Luckily” I only had a few paragraphs done. It’s too bad there’s not a tool like Xjournal that can interface with the O’Reilly weblog interface. Guess I’ll use TextEdit for now.

Which do you use: Fink or DarwinPorts? Have any cool tools you like to use that are available through either?

I run an intranet and a staging server on non-standard ports (8000 and 8080). This works great for our internal purposes, but every now and then a student will want to show a work-in-progress to an external organization. And every now and then, that organization turns out to be living behind one of those Stalinist corporate firewalls that blocks everything but port 80, which means they can’t access the content, which means the student comes to me baffled, I explain the situation, and no one understands what I’m talking about. Somehow it always comes off as if I’m the one blocking the traffic. Ports are hard to explain to non-tech people. If I ask them to ask their sysadmins to back off a bit and open up traffic on these ports, I always get the same “we don’t do that for security reasons.”

Does it really make security sense for organizations to blindly block everything but port 80? The internet runs on ports. It’s all about ports. There’s got to be a more sensible way to accomplish your security goals than to slam the door in the face of other services. Are they being paranoid or am I expecting too much?

How have you dealt with this situation?

It’s been a busy few days at the Software Update corral. First Mac OS X 10.2.5, then Safari Beta 2. There’s lots of discussion about the former, so I want to give you a brief overview of my three favorite features in the latest version of Safari.

What you’ll hear the most about in this beta are tabbed browsing, AutoFill forms, and Privacy Reset. Even though tabbed browsing is helpful, especially for those with 12″ screen laptops, I really have been waiting for AutoFill, and the Privacy Reset is a pleasant surprise.

After you download the new beta via Software Update or from the Safari page, go directly to Safari’s preferences. You have a few new items to adjust.

First stop is Tabs. Check the enable box and decide on which options you want. One cool thing Apple does here is give you the keyboard commands for controlling Tabs right there in the preference window. Basically, the tabbing works as you’d expect. One clever touch that tickles me is the little spinning “loading wheel” that occupies the right side of the tab telling you that the page is still rendering. Cute.

Next, go to AutoFill (Yaay!). Gosh I’ve missed AutoFill. Safari uses your “Me” card in Address Book for its AutoFill. So I recommend that you click the Edit button and make sure your info is the way you want it. Generally speaking Safari uses the top entry in any given category to complete the form. So, arrange your various email addresses accordingly. Then when you encounter a form, the command to fill it is under the Edit menu. There’s also a new button on your menu bar, as well as a new keyboard shortcut if you want to use that: SHIFT/CMD/A

But wait, there’s more. You can also have your web accounts and passwords available (stored securely in your Keychain). What a terrific way to help manage all those secure sites that have become a big part of our online life.

But I wouldn’t be quite as pleased about the password feature had Privacy Reset not been added too. The feature is included under the Safari menu as “Reset Safari…”. When you choose it, everything gets wiped: history, cache, downloads, Google search entries, cookies, saved names, passwords, and AutoFill text.

Now that’s thinking. So even though I welcome tabbed browsing with open screen real estate, AutoFill and Privacy Reset are my personal favorites here.

I know, the J-School should be all paneled out by now, not to mention weblogged out, but lo, another good webcast tonight:

Weblogs are mainstream, and they are changing the way we manage knowledge, work and communicate. On Thursday, April 10, join panelists Dan Gillmor, Scott Rosenberg, Donna Wentworth and others at the J-School in exploring how this change continues to affect academia, journalism, business, and society.

Just upgraded QuickTime Streaming Server to v4.1.3, and the initial load latency has been drastically reduced — no more 5-7 seconds of buffering before stream starts. Not quite sure how they accomplished that, but it’s impressive.

There’s a touching scene in the Minority Report where Chief John Anderton (Tom Cruise) gets stoned in his high-tech apartment and takes a visual trip down memory lane.

He doesn’t look at snapshots or read old letters. Instead, he pops a video chip in a holographic projector and watches home movies from his lost family life. The ghost like figures seem so real, so near, yet they are nothing more than memories projected into thin air.

This week, the National Association of Broadcasters (NAB) is holding its annual conference in Las Vegas, NV. If you’ve been to this show in recent years, you know that it covers just about anything that moves in the world of media. And the world of media is indeed moving closer to the hands of serious enthusiasts.

For example, Apple unveiled a new version of Final Cut Pro. Among its breakthroughs, FCP now has native 24-framesupport for Panasonic’s AG-DVX100. In short, this represents a step closer to producing film-like big screen movies on the home desktop. The Panasonic camera was a breakthrough announcement a while back and made a big splash at the Sundance Film Festival.

As Hollywood production moves into the home studio, consumer-level tools are becoming easier to use too. January’s announcement of iMovie 3, Final Cut Express, and iDVD 3 mean that motivated hobbyists can create professional-looking DVDs with a Mac and a $500 DV camcorder. But I think “can” is still the operative word here. Just because you “can” doesn’t mean that you will.

Maybe we are moving in the direction that Speilberg portrayed in the Minority Report where life-like video serves as our personal link to the past, our family history, and even our memory itself. Moving toward… maybe. But not quite there yet.

Until consumer-level DV camcorders get cheaper and better at recording life-like sound (you still have to use an external mic for quality), and until hard drives get a helluva lot bigger and cheaper too (you need a whopping 13 GBs for one hour of digital video), decent DV libraries will remain on the shelves of the serious enthusiast and not the casual user.

Cell phones are ubiquitous communications tools because everyone can use them well (manners notwithstanding). Digital still cameras are becoming more popular because the prices have dropped and people realize that they can take better pictures with them than with their film counterparts.

But for the moment, good digital video still requires a modest equipment investment and some technical savvy. I want DV to be easier because I think it’s a valuable communication tool, historical recorder, and lots of fun. I want to see the day when we use video to record our lives the way our grandparents wrote letters and told stories — not instead of the other media, but in addition to.

I’m working on a pocket guide right now to help make shooting video as easy as possible. But the tools still need to improve for the average consumer. Hopefully in the next year or two, mass market video will see the same kind of progress we’ve seen recently in digital still photography. We’re getting close, and it’s exciting.

For the technically-minded, now seems like a good time to jump in. The computer tools are interesting, somewhat affordable, and produce solid results. If you’re interested, take a look at what’s happening at the NAB show in Las Vegas. It really is amazing.

Now, about those home holographic projectors…

I’ve talked about Apple’s Rendezvous technology in the past. I strongly feel that it is changing the face of networking as we know it. Setting up a functional network with basic services is no longer the realm of the geekier among us. Rendezvous empowers the common man. Today I’m going to highlight three Mac OS X applications that make excellent use of Rendezvous.

Hydra

Hydra is a Rendezvous aware text collaborative text editor. It allows a user to share plain text files over a link-local or IP network easily. It makes use of Rendezvous to help you discover other shared documents in your local area network. Shared documents appear in Hydra’s Rendezvous browser, complete with the user icon and name of the person sharing the file.

Joining a document is as easy as double clicking it in the Rendezvous browser. The person sharing the document then approves your request to access it. You can then make changes to the document in realtime, with Hydra handling contention as well as highlighting the changes made by each user. Hydra also features syntactical highlighting for several different markup and coding languages and works over an IP network, allowing users to work together even if they are not on the same local area network.

EvoCam

EvoCam is a webcam application that features several different output methods for your captured images. With the release of version 3.1, Evological has added a Rendezvous advertising web server that allows your webcam to be viewed from any browser that detects Rendezvous servers. Apple’s Safari and Camino both support Rendezvous, allowing users to easily connect to your cam.

In addition to Rendezvous serving, EvoCam features the ability to record both still images and video, can handle multiple cameras simultaneously, and allows you to add logos and text to your captures. It’s everything you could ever want in a webcam app and then some. One of the coolest features is its AppleScript support, which allows you to create scripts that will be run during certain EvoCam events. I have one which queries iTunes and lists my currently playing track on a text overlay element.

iConquer

The last app I’m going to discuss is KavaSoft’s iConquer. I’ve always been a Risk fan and am impressed with this Risk-like game’s creative AI and its convenient interface. Throw in the support for Rendezvous based gaming and it’s a clear winner. KavaSoft has taken a classic board game and brought it to the Macintosh platform in a unique and enjoyable way.

More in the Pipeline

These are just three applications that I’ve enjoyed using recently. I’m already looking forward to the new applications that are being developed with Rendezvous in mind. I’m especially hoping for a good IRC application that can detect Rendezvous chat rooms. I imagine such a program would be a huge hit at the various Mac expos. These programs are just the beginning. Rendezvous… it’s the future baby ;)

What are some Rendezvous aware apps that you enjoy using?

Related link: https://www.oetrends.com/cgi-bin/page_display.cgi?194

“The first book on the inside operations of Rotor, Microsoft’s Shared Source CLI (Common Language Infrastructure) has been released this month by O’Reilly. To get a close up view of how integration and Open Source developers (and academic researchers/designers) can leverage the CLI, OET spoke with co-author David Stutz, a member of the core design team for the .Net Framework, where Stutz also served as a technical strategist and software architect at Microsoft.”