A minimalistic yet much more powerful bookmark manager, kicking del.icio.us' and Firefox/you-name-it's ass

Barely sub-second latencies? Poor searching capabilities and on top of that you have to wait several RTTs and use the mouse?! I never managed to force myself to use del.icio.us. Actually, I didn't really use any bookmark manager; those from the several browsers I run all suck. I just remembered pieces of URLs...

But this one I'm liking so far: based on wmii's wmiimenu+wmiipsel tools, and built on top of ruby-wmii, it features:

• mouse-less interaction
• search as you type (extended autocompletion) for both title and URLs: the set of bookmarks matching what I'm typing at any position in the title or the URL is updated instantaneously as I type
• del.icio.us integration: importing bookmarks from del.icio.us and getting new ones automatically
• tagging (it will import your del.icio.us tags if you let it try)
• powerful search expressions (as many criteria as you want):
  • all bookmarks in the last week: ~d <7d
  • all bookmarks whose description matches a regexp: ~t regexp
  • all bookmarks with "redhanded" on the description or the URL, defined/last used in the last month: redhanded ~d <1m
  • all bookmarks with "ruby" on the URL, defined/last used in 2006: ~d 2006 ~u ruby
  • all bookmarks tagged as "blog", defined/last used in Q1: :blog ~d q1
• progressive refining: I can enter successive expressions and each one further restricts the possible choices, which are shown in the menu

I have added the BookmarkManager to ruby-wmii's standard plugin in the development branch. It stores the bookmarks as a plaintext file in $HOME/.wmii-3, so you can use the text tools of your choice to search and process it.

Here are some animations showing how it works:

Progressive refining

First the bookmarks matching 'red', then the remaining ones under 20 days old, finally the subset of the latter with wmii, no, hpricot somewhere in the description or the URL:

Here I first restrict the choices to bookmarks tagged as :wmii, then pick amongst those with "red" either in the description or the URL:

Search as you type

As usual it must be activated in wmiirc-config.rb:

from "standard" do
  use_binding "bookmark"      #, "MODKEY-x"      uses MODKEY-Shift-b by default
  use_binding "bookmark-open" #, "MODKEY-y"      uses MODKEY-b by default
end

# if you want the del.icio.us sync'ing:
# plugin_config["standard:bookmark"]["del.icio.us-user"] = "username"
# plugin_config["standard:bookmark"]["del.icio.us-password"] = "password"

You can bookmark an URL by placing it in X11's selection buffer (either select with the mouse or yank from vim ;) and pressing MODKEY-Shift-b. The page will be fetched and a wmiimenu will open, allowing you to set the description; a few possible completions will be generated based on the title. In order to tag the bookmark, just append :some :tags to the description (that is, tags look like symbols in Ruby).

MODKEY-b will open the bookmark menu. You can pick the desired bookmark with any combination of cursor movements (that choose amongst possible completions), normal typing (which matches against the URL or the description) and complex expressions. If more than one choice matches the expression you entered, a new wmiimenu with the remaining candidates will be shown, until you select a single one.

See the original implementation (much simpler, but still better than nearly everything out there) here.

Some code

require 'time'
require 'thread'
class BookmarkManager
  Bookmark = Struct.new(:description, :url, :tags, :date)

  def initialize(filename)
    @filename = filename
    @bookmarks = []
    @bookmark_index = Hash.new{|h,k| h[k] = {}}
    @loaded = false
    @mutex = Mutex.new
  end

  def load
    @bookmarks = []
    @bookmark_index.clear
    IO.foreach(@filename) do |line|
      desc, url, tags, date = line.chomp.split(/\t/).map{|x| x.strip}
      tags = (tags || "").split(/\s/)
      begin
        date = Time.rfc822(date)
      rescue
        date = Time.new
      end
      bm = Bookmark.new(desc, url, tags, date)
      @bookmarks << bm
      @bookmark_index[desc][url] = bm
    end rescue nil
    @loaded = true
  end

  # Returns the bookmark if unique.
  def [](desc)
    self.load unless @loaded
    return nil unless @bookmark_index.has_key?(desc)
    bms = @bookmark_index[desc]
    if bms.size == 1
      bms.values[0]
    else
      nil
    end
  end

  # Returns true if it was a new bookmark, false if the (desc,url) was not
  # unique or it was older than the existent.
  def add_bookmark(desc, url, tags, date)
    self.load unless @loaded
    ret = true
    if @bookmark_index.has_key?(desc) && @bookmark_index[desc].has_key?(url)
      return false if @bookmark_index[desc][url].date >= date
      @bookmarks.delete @bookmark_index[desc][url]
      ret = false
    end
    bm = Bookmark.new(desc, url, tags, date)
    @bookmarks << bm
    @bookmark_index[desc][url] = bm
    ret
  end

  def bookmarks
    self.load unless @loaded
    @bookmarks
  end

  # This method is thread-safe, not process-safe.
  # It will merge the bookmark list with that on disk, avoiding data losses.
  def save!
    @mutex.synchronize do
      merge!
      tmpfile = @filename + "_tmp_#{Process.pid}"
      File.open(tmpfile, "a") do |f|
        @bookmarks.sort_by{|bm| bm.date}.reverse_each do |bm|
          f.puts [bm.description, bm.url, bm.tags.join(" "), bm.date.rfc822].join("\t")
        end
        f.sync
      end
      File.rename(tmpfile, @filename) # atomic if on the same FS and fleh
    end
  end

  def merge!
    IO.foreach(@filename) do |line|
      desc, url, tags, date = line.chomp.split(/\t/).map{|x| x.strip}
      tags = (tags || "").split(/\s/)
      begin
        date = Time.rfc822(date)
      rescue
        date = Time.new
      end
      add_bookmark(desc, url, tags, date)
    end rescue nil
  end
  private :merge!

  def satisfy_date_condition?(bookmark, condition)
    date = bookmark.date
    case condition
    when /^q1$/i : date.month >= 12 || date.month <= 4
    when /^q2$/i : date.month >= 3  && date.month <= 7
    when /^q3$/i : date.month >= 6  && date.month <= 10
    when /^q4$/i : date.month >= 9 || date.month <= 1
    when /^\d+$/ : date.year == condition.to_i
    when /^\w+$/ : date.month - 1 == Time::RFC2822_MONTH_NAME.index(condition.capitalize)
    when /^([><])(\d+)([md])/
      sign, units, type = $1, $2.to_i, $3
      multiplier = 3600 * 24
      multiplier *= 30.4375 if type == 'm'
      case sign
      when '<':  Time.new - date <= units * multiplier
      when '>':  Time.new - date >= units * multiplier
      end
    end
  end
  private :satisfy_date_condition?
  
  def refine_selection(expression, choices=self.bookmarks)
    expression = expression.strip
    pieces = expression.split(/\s+/)
    criteria = []
    option_needed = false
    pieces.each do |x|
      case option_needed
      when true:    criteria.last << " #{x}"; option_needed = false
      when false:   criteria << x; option_needed = true if /^~\w/ =~ x 
      end
    end
    choices.select do |bm|
      criteria.all? do |criterion|
        case criterion
        when /~t\s+(\S+)/: Regexp.new($1) =~ bm.description
        when /~u\s+(\S+)/: Regexp.new($1) =~ bm.url
        when /~d\s+(\S+)/: satisfy_date_condition?(bm, $1)
        when /:\w+$/     : bm.tags.include?(criterion)
        else bm.description.index(criterion) or bm.url.index(criterion)
        end
      end
    end
  end
end

syncing with del.icio.us - olli (2006-08-28 (Mon) 08:05:54)

Syncing with del.icio.us only works onedirectional, right? I'd appreciate it if they'd be bidirectional (so that new local bookmarks would automagically be uploaded (marked as not to be shared, for privacy reasons)), as then I could make use of the possibility to add bookmarks at work and use them at home and vice versa. (I want ubiquitous bookmarks now!)

Again many thanks for your great work!

mfp 2006-09-10 (Sun) 13:55:17

Dmitry Kurochkin & I have been working on this. The branch in the online darcs repository supports bidirectional sync'ing already but we're still refining it.

olli 2006-09-16 (Sam) 13:47:51

I gave it a try tonight. First, I ran into the problem that uploading to del.icio.us did not work because of a nonexistant booksmark.txt.remote file, which you try to delete, regardless whether it exists on the system or not. So in »update_remote_delicious_bookmarks« the line

       File.delete BOOKMARK_REMOTE_FILE

has to be changed to

       if File.exist?(BOOKMARK_REMOTE_FILE) 
         File.delete BOOKMARK_REMOTE_FILE
       end

Then syncing works for me. However, I get problems with german umlauts, which aren't displayed properly in del.icio.us. (Importing did work with no problems).

Thanks

mfp 2006-09-16 (Sat) 15:13:01

I pushed a couple patches from Dmitry a few hours ago that should fix both issues.

There's a new configuration option to specify the encoding used locally (probably latin1 in your case):

plugin_config["standard:bookmark"]["encoding"] = "ISO-8859-15"

Here's some extra info I added to the default wmiirc-config.rb:

 ## Sets the encoding used to:
 #  * store the bookmark descriptions in bookmarks.txt
 #  * present choices through wmiimenu
 # Please make sure your bookmarks.txt uses the appropriate encoding before
 # setting the next line. If you had already imported bookmarks from
 # del.icio.us, they will be stored UTF-8, so you might want to
 #   recode utf-8..NEW_ENCODING bookmarks.txt
 #
 # If left to nil, bookmarks imported from del.icio.us will be in UTF-8, and
 # those created locally will be in the encoding specified by your locale.

By the way, the main development branch has moved to http://eigenclass.org/repos/ruby-wmii/head/

olli 2006-09-16 (Sam) 17:52:01

I did so, but ran into some problems, as some descriptions of my del.icio.us bookmarks made use of utf-8-chars. E.g. this one: »Frank Mittelbach - Formatting documents with floats. A new algorithm for LaTeX 2ε« <http://www.tug.org/TUGboat/Articles/tb21-3/tb68mittel.pdf> »floats« starts with a ligature (»fl«) and there's an »ε« in »LaTeX 2ε«. The script stumbled over these and broke.

It worked after I manually changed the description on del.icio.us. However, this bookmark was then deleted at del.icio.us. I rebookmarked it and now everything seems to work fine.

mfp 2006-09-19 (Tue) 03:00:18

I'm telling Dmitry about this & looking into it, thanks for trying it out.

thrilling!! - _why (2006-07-09 (Sun) 22:13:27)

walloop! such a killer app, this is so ready to be my closest friend... ohoh... i can't believe what fun.

Twinks for Cash 2006-11-21 (Tue) 00:35:27

Very good site. Thank you!!! http://myblog.es/sisers/ Twinks for Cash http://myblog.es/sisers Young Twinks for cash

plugin_config inside from-block - cmarcelo (2006-07-08 (Sat) 15:47:23)

I got error here when use plugin_config inside from-block. Just took it outside and worked fine.

mfp 2006-07-08 (Sat) 16:18:50

oops, right, plugin_config must be outside the from "standard do ... end block

BTW I forgot to say that once you set del.icio.us-user and del.icio.us-password bookmarks get sync'ed every 30 minutes by default, but you can override that with e.g.

 plugin_config["standard:bookmark"]["refresh_period"] = 60

to refresh once an hour.

One last thing: if you want to import all the bookmarks (you'll do that only once, at the beginning), you can use the del.icio.us-import action (MODKEY-a -> del.icio.us-import).

mardoen 2006-07-08 (Sat) 19:45:14

(Hint: animated GIFs are not a good medium for this kind of animation. I'd like to have an option to pause and read individual steps. Better in this respect: video/screencast.)

mfp 2006-07-09 (Sun) 04:49:05

Agreed; unfortunately I don't have a Ruby script to generate them, as I have for gif animations :) Ultimately, the only way to get the feeling of these UI things is giving them a try.

olli 2006-08-27 (Son) 05:40:06

Am I wrong or is it not possible to add https-sites to the bookmarks? I don't have any knowledge of ruby so I don't feel to be able to change the script, but you just test for `http://'. Is there any deeper meaning of this or just not (yet) implemented?

P. S.: Sorry for thread hijacking, but I can't start a new one. Perhaps your script is broken? I see a `NameError (undefined local variable or method `bbs_text'' above.

P. P. S.: I even get an error when clicking on preview: `undefined method `make_anchor' for nil:NilClass Please back to MAIN.' I'm trying anyway.

mfp 2006-08-27 (Sun) 14:03:14

As for the URL regexp, I just forgot to consider HTTPS. I've just pushed a patch that solves this and sets the User-Agent (some sites refuse to serve the page otherwise) for the request. You can find the latest code in the darcs repository at http://eigenclass.org/repos/ruby-wmii

As for the brokenness in eigenclass.org, I think I just fixed the two bugs you found.

The first one (undefined bbs_text) was due to a missing translation, I think (you're using de in HTTP_ACCEPT_LANGUAGE, right?); for the time being I just copied the English description.

I had bumped into the second bug a couple times before but never managed to obtain a stack trace... I was lucky this time and it should work now.

Thank you!

olli 2006-08-28 (Mon) 07:47:37

Uhh, thank you for your work! I'm heavingly using ruby-wmii all the time and even thinking of learning ruby. Bookmarking https-sites works now, and so do the form for top level comments (yes, I am a german user and have configured my browser to prefer the german language) and the preview :-)