Mail AppleScript-ing Project

I have a folder full of old mail in Apple’s Mail application. It’s gigantic. About 70,000 messages. Most of them are duplicates, because it’s the result of finding old folder of mail upon old folder of mail and merging them together into one great hoard. The actual number of real distinct messages is probably a smallish fraction of 70,000.

What’s worse, some of the 70,000 are blank. In an inept attempt at writing a Python script to clean up a similar uber-mail-folder in the past, I somehow took a lot of old mail and destroyed the body of the emails, leaving the headers intact. So my gigantic folder includes many duplicates, but some of the duplicates aren’t real duplicates because they have missing bodies.

I want to somehow eliminate all the duplicate messages, and there are scripts to do that in Apple Mail. The only one that I would have trusted not to accidentally kill a real message and keep the one without a body, chokes and fails on a folder that large. (It also choked and failed on a smaller folder. Maybe something changed in Leopard that breaks that script.)

So I wanted to go through and destroy all the messages which have blank bodies — they’re no use to me and they make it dangerous to get rid of duplicate messages. I tried exporting everything to a mbox-format file, and use some of Python’s nice mailbox-manipulation libraries, but the file was insanely large, and Python on my macbook staggered under its weight. (Besides, my use of Python caused this problem, a while back…)

So eventually I turned to AppleScript. (I first tried using rb-appscript, but it turns out I don’t need any special Rubyness for this, and it’s easier to learn from examples of AppleScript on the web if I don’t have to translate them into Ruby before I use them.)

I wrote a script in Apple’s Script Editor called “Winnower.” It takes messages in a folder called “doing” and sorts them into two folders, “blank” and “done,” depending on whether there’s any content in the body or attachments on the mail. I put a few thousand messages at a time into the “doing” folder and then run the script. (The full weight of the 70,000+ message folder was too much for this script too.)

It looks like this:

tell application “Mail”

set doingbox to mailbox “doing”

set blankbox to mailbox “blank”

set donebox to mailbox “done”

set doingmessages to messages of doingbox

repeat with thisMessage in doingmessages

ignoring white space

if mail attachments of thisMessage is {} and content of thisMessage is equal to “” then

move thisMessage to blankbox

else

move thisMessage to donebox

end if

end ignoring

end repeat

end tell

Hardy Heron Saves The Day, Sort Of.

Yesterday the kids/family’s computer’s hard drive took a dive. (It’s a very old desktop machine which my wife bought from her employers for something like $10 when they upgraded all the machines; that was two or three years ago. It has always run Ubuntu Linux in our house.) Wouldn’t boot, couldn’t even hit the grub menu. Wouldn’t be so bad but the hard drive was big and I’d been using it as a backup drive for a lot of stuff. I have a new backup system (a portable HD that I back my macbook up to) but I still had a lot of old stuff on there that I would like back. I haven’t yet looked into whether it can be resurrected. I took it out and swapped in an old small drive.

Since there’s a spanking new version of Ubuntu available, Hardy Heron, I burned a CD of that and installed it. Install went smoothly but the graphics have been killing me.

There were two problems, one now solved:

  • There is no way to get past 800×600 resolution even though I know that it’s capable of much better
  • The graphics were *weird* — when you moved the mouse pointer you’d get little messed up squares on the screen where the mouse went. (Now solved.)

The problem at the root of all this is that they have kicked autoconfiguration of X11 into overdrive. It is no longer possible to manually configure your graphics card and monitor set up by any straightforward means. It autoconfigures, and the configuration file which it was once possible to tweak is now a nigh-empty stub. (It’ll still respect what you put in there, but I don’t know enough off the top of my head to do anything with it.)

The causes of the two above-mentioned problems come from this autoconfiguration:

  • The autoconfigurator doesn’t seem to know what my monitor is capable of, and assumes the worst (800×600)
  • The autoconfigurator is smart enough to be able to take advantage of my card’s hardware acceleration, and so tries to run a cool whizzbangy hardware-accelerated desktop. Problem: my card is old and crappy. Its hardware acceleration is bad. I do not want a cool whizzbangy hardware-accelerated desktop, because of that.

I was able to turn off the whizzbangy desktop effects which ruined the desktop, through the Appearance item in the Settings menu. I haven’t yet figured out what to do about the resolution. We’ll see how that goes tonight.

On the upside, everything else about the machine is Just Working.

A Tasty Video Recipe

I like to download videos from youtube using the magically delicious DownloadHelper. Especially music videos. The only sad thing is that the video quality tends to be poor, and it can only get worse if you convert it to a less crappy format than the standard flv — such as mp4 for example.

But Youtube is being kind to us. Many, many videos on youtube, if not all, are available in a higher quality format — in fact, mp4, the format that is standard for things like iPods. The magic you do to get the higher quality is take the youtube url and add “&fmt=18” to it. That’s all. Just add that to the address in the address bar and reload and you will get a higher quality video, which you can then download with downloadhelper. You might need to change the .flv extension to .mp4 to play it back with Quicktime.

There’s also a thing in Youtube’s account preferences you can set to get higher quality stuff by default, but I don’t think it’s the same — I think (not sure) it gives you the fmt=6 version. (there are at least two different higher quality settings, &fmt=6 and &fmt=18 — I think the &fmt=6 is better than the default but not mp4-encoded like &fmt=18.)

There’s a greasemonkey userscript to do the fmt=18 for you.

UPDATE: if you see something in the downloadhelper.net menu called Video.mp4 that’s guaranteed to be the good stuff. I think it also shows up under other names sometimes too — downloadhelper sometimes offers several links to the same video so it’s confusing.

UPDATE: it ain’t just video. The audio is *noticeably* better in the higher quality downloads. Much crisper, makes the old kind sound muddy in comparision.

Three Awesome Drawing Apps For OS X Under $30

I’ve discovered three really outstanding, and really reasonably priced, shareware drawing applications for OS X. If you’ve blown your budget on a Wacom pad and suddenly realize you can’t afford something huge and crazy like Painter or Photoshop — and you discover that the Gimp for OS X isn’t hip to pressure-sensitive tablets — check these out.

ArtRage from Ambient Design

My homeboy Adam Black pointed me at this one, I think. It’s cross-platform. It’s kind of like Corel Painter in that it’s oriented towards simulating natural media — chalks, oil paints, pencils, watercolors, crazy things like that. Development is ongoing; it’s been updated a couple times since I bought it. There’s a reasonably functional free version as well.

What’s Awesome: It is really, really, REALLY good at simulating natural media. You can seriously create things that look exactly like chalk drawings on textured paper, or oil paints on canvas, or whatever. It takes a while to learn to control the media, but their forums are really helpful. It has layers, “stencils,” “reference images,” all kinds of crazy things. There is an immense amount of power in Artrage.

What’s Less Awesome: Ironically, it’s kind of hard to make drawings that don’t yell “NATURAL MEDIA!!!!” at the top of their lungs. Getting smooth paper and a simple black line is actually a bit of a hassle. The interface, while really original and cool, seems not to have been originally designed for macs, and if you go fullscreen it kind of fights with the Dock. (So I don’t.)

TabletDraw from mooSoftware

This is seriously minimalist. It is aimed at doing one thing very well: drawing with a tablet.

What’s Awesome: They claim to have an outstanding algorithm/whatever for accurately recording quick, subtle strokes, and my experience bears that claim out. You can sign your name quickly on the tablet and it will look just right on the screen. You can create really nice line drawings with it; you can tweak the “pens” along several axes (size, how much they grow/shrink depending on pressure, transparency, color, and whether they allow darker lines underneath them to show through, which is called “ink mode”). It’s easy to build up a palette of useful brushes. You can do interesting things with it once you learn to work with its minimalism. Essentially everything you do with it is “pen drawing” — but you can have “pens” with any color ink, any transparency, any size, the ability to layer over darker ink, etc.

What’s Less Awesome: The minimalism is extreme. They have no layers, for example. A recent feature is that you can load a reference drawing up in the background, and draw on “onionskin” above it, which is a large fraction of what you’d want to do with layers, but that is unwieldy — you have to save an image and then reload it as a reference. The resizing behavior of canvas vs. window is weird too until you learn how to work with it. These are not dealbreakers by any means, but TabletDraw is something that you have to learn your way around. It’s not always what you expect it to be, and it’s not everything to everybody.

Scribbles from Atebits Software

I’ve only just discovered this one. Scribbles is so slick it looks like it was created for Steve Jobs to demo at a keynote address. It uses the latest Mac imaging technology (so you need OS X 10.4 or better to use it), and it’s — wow.

I haven’t had long to explore it but here are first impressions:

What’s Awesome: Incredibly slick, easy to use interface. All the controls and tools are giant and easy to select. The layers control shows animated layers in perspective and how they’re related to each other. Responsiveness to tablet pressure and movement is good. There are a good number of basic tools (pens with varying degrees of fuzziness, erasers, some wacky coloring tools).

No, Really, What’s Awesome: Resolution-independent infinite canvas. You are drawing on an infinte sheet of paper and can zoom in or out all you want, for an arbitrary level of detail or expansiveness. It’s kind of like Rita (also worth checking out, and free) in that way, but I’m finding it a bit easier to use than Rita. In the other programs I’ve used, I usually create a very large canvas to start with so I don’t have to worry about pixellation… I don’t have to do that here.

More Awesome: “tracing paper” mode, where the *entire application* turns semitransparent, so you can trace, say, an image in your web browser.

What’s Not So Awesome: Haven’t discovered that yet but I’m sure I will. Nobody’s perfect. I guess you can’t use it to do simulated natural media like in ArtRage at all, and I don’t know whether its tablet movement responsiveness is as exquisitely awesome as TabletDraw’s. It doesn’t seem to have the ability to really customize your pen tips to be just what you want, like in TabletDraw, so if you want to do exquisite penwork TabletDraw is probably still the winner. But man. It’s tres slick.

Any one of these is easily worth your money. Check ’em out if you’re in the market.

Stupid Ruby Serialization Tricks

This post is for the small fraction of readers of this blog that dig programming in Ruby… apologies to anyone else….

So, I was fooling around with Ruby when I came up with the following trick.

Let’s say you have an object that you want to be persistent across invocations of your application. A set of configuration settings, history, who knows what. You don’t want to go nuts and worry about a whole database though. There are a couple good super-light-weight transactional persistence libraries built in to Ruby, PStore and YAML::Store, which do the job. You could them like this:

o = MyCoolObject.new

# to store it first
YAML::Store.new(".storage_for_my_app").transaction do | store |
  store['MyCoolObject'] = o
end

# .. and to retrieve it from storage.
YAML::Store.new(".storage_for_my_app").transaction do | store |
  o = store['MyCoolObject']
end

You’ve got to remember to store it again when you’re done with it of course.

I came up with a variation on this:

class MyCoolObject
  def MyCoolObject.stored_in(filename, *newargs)  # class method
    YAML::Store.new(filename).transaction do | store |
      store['self'] ||= MyCoolObject.new(*newargs)
      yield store['self']
   end
  end
end

Now I can make sure everything I do with my cool object “o” is stored persistently. All I have to do is wrap my actions on “o” in a stored_in block:

MyCoolObject.stored_in(".storage_for_my_app", 'a', 'b') do | o |
  # o is either recreated from .storage_for_my_app, or
  # created anew with args 'a' and 'b' passed to its constructor

  # .. here we do stuff with o
end
# and here o is re-serialized, with any changes intact.

When the block opens, ‘o’ is either resurrected from the storage file, or created anew from *new_args if there’s nothing in the file.

When the block ends, any changes to ‘o’ are stored there.

As long as you keep all your interaction with the object inside stored_in blocks, you’re golden. You get persistence!

Note that ‘o’ can hold other objects in its instance variables, which can hold other objects, and so on — anything in there that’s serializable can be stored this way. So you can use this to persist a whole pile of objects in one file as long as they all live in a single object which has a stored_in class method.

I thought this would be cool functionality to include in a module, whereupon I learned that in Ruby, a module’s “class methods” are not added to a class when you include a module in it. (I guess because a module isn’t a class and so it doesn’t really technically have “class methods”…) but you can get the same effect using a tiny bit of trickery with the “included” method of the Module class.

Here’s a module you can use to give any class these kind of storage abilities:

require 'yaml/store'

module StoredInFile
  def self.included(base)
   def base.stored_in(path, *args)
     YAML::Store.new(path).transaction do | store |
       store['self'] ||= self.new(*args)
       yield store['self']
     end
    end
  end
end

You use it like this:

class MyCoolObject
include StoredInFile

# other class stuff goes here

end

And that’s it.

UPDATE:

Stupid WordPress makes including code in a page really hard…  keeps eating my formatting.  I think I’ve got it…