eigenclass logo
MAIN  Index  Search  Changes  PageRank  Login

When the GC is doing its job, but your app still needs too much RAM

Sometimes your code is using much more RAM than it should. You've made sure that you are not keeping too many large objects around, you got rid of their references and you are sure they were GCed. Yet ps is saying your code takes up too much memory. Memory fragmentation can be nasty at times, but you can avoid it by allocating carefully.

Making sure our objects are being collected

Before you blame fragmentation, you've got to make sure the GC is doing its job properly (and you're helping it do it). The obvious way is iterating over the ObjectSpace and counting how many objects of each class there are. I've seen that rewritten countless times :)

Estimates of the amount of memory used for each kind of object are a bit less common. I don't mean, say, somestring.size, but something a bit more accurate that takes into account the overhead introduced by ruby's internal structures.

This is taken from FastRI; I was mostly concerned about Arrays there, but it can be easily completed with the information from the above link:

if $DEBUG
  # trying to see where our memory is going
  population = Hash.new{|h,k| h[k] = [0,0]}
  array_sizes = Hash.new{|h,k| h[k] = 0}
  ObjectSpace.each_object do |object|
    # rough estimates, see http://eigenclass.org/hiki.rb?ruby+space+overhead
    size = case object 
           when Array
             array_sizes[object.size / 10] += 1
             case object.size
             when 0..16
               20 + 64
             else
               20 + 4 * object.size * 1.5
             end
           when Hash;    40 + 4 * [object.size / 5, 11].max + 16 * object.size
           when String;  30 + object.size
           else 120 # the iv_tbl, etc
           end
    count, tsize = population[object.class] 
    population[object.class] = [count + 1, tsize + size]
  end

  population.sort_by{|k,(c,s)| s}.reverse[0..10].each do |klass, (count, bytes)|
    puts "%-20s  %7d  %9d" % [klass, count, bytes]
  end

  puts "Array sizes:"
  array_sizes.sort.each{|k,v| puts "%5d  %6d" % [k * 10, v]}
end

This will tell you how much memory your arrays/hashes/strings/other objects are taking. It could be modified to proceed recursively, so the space needed for objects held in instance variables is attributed to the parent object, but it might not be doable if there are multiple references (you either count it only once, possibly in the wrong object, or repeatedly).

Fighting memory fragmentation

Here's a simplification of the indexing loop in the first version of FTSearch full-text search engine (of course, it didn't quite look like this, work was divided more cleanly among the fulltext store, the document map and the suffix array writer):

file_list = Dir["corpus/linux/**/*.{c,h}"]
documents = {}
file_list.each do |file|
  documents[documents.size] = {:uri => file, :body => File.read(file)}
end
suffix_array = []
documents.each_pair do |doc_id, doc_hash|
  document_map.add_document(doc_id, doc_hash[:uri], size_of_doc(doc_hash))
  doc_hash.each_pair do |field_name, data|
    # save data to disk
    offset = fulltext_store.store(doc_id, field_name, data)
    analyzers[field_name].append_suffixes(suffix_array, data, offset)
  end
end

documents.clear  #
documents = nil  # make sure we get rid of all the references
GC.start         # (in practice, all this is wrapped in a method and we try 
                 # to make sure there are no references left in the C stack 

fulltext = fulltext_store.fulltext  # this loads the data from disk

sort(suffix_array, fulltext)  # entries in the suffix_array are offsets into 
                              # the fulltext
dump(suffix_array)

All it did was collecting the data to be indexed in the 'documents' hash, save it in the fulltext store while suffixes (offsets into the fulltext) are collected, discard the 'documents' hash, reload the data from the file where the fulltext was saved, sort the suffixes and dump them to disk.

How much mem should it have taken? In theory, when sorting it would need:

  • the space needed to hold the data (in the fulltext string)
  • the space taken by the suffix array (sorting is in-place and doesn't take much more)

The documents hash is GCed before the fulltext is read from disk, so it won't take twice the amount required to hold the data itself, in theory.

When I used that to index all the .c and .h files in Linux' source tree (with ~160MB of text, and 20 million suffixes), I could have expected around 160 + 20 times 4 + something approx 250-270 MB

But it took over 400MB. The reason: free() wasn't able to return memory to the system. Some objects were created after the 'documents' hash was built, and when it was reclaimed lots of holes were left in memory (I'd verified that the 'documents' hash had been collected.)

The solution: finding suffixes as we read documents and saving to the fulltext store incrementally. This is slower IO-wise, since the HD will have to alternate between reading from a file and writing to the fulltext, but it doesn't make that large a difference in practice (about ~15s or so to index Linux' sources, which takes 2 minutes overall), and avoiding swapping takes priority.

So this, which looks quite similar to the above code, will take much less memory:

file_list = Dir["corpus/linux/**/*.{c,h}"]
num_documents = 0
file_list.each do |file|
  doc_id = num_documents
  doc_hash = {:uri => file, :body => File.read(file)}
  document_map.add_document(doc_id, doc_hash[:uri], size_of_doc(doc_hash))
  doc_hash.each_pair do |field_name, data|
    # save data to disk
    offset = fulltext_store.store(doc_id, field_name, data)
    analyzers[field_name].append_suffixes(suffix_array, data, offset)
  end
  num_documents += 1
end

fulltext = fulltext_store.fulltext  # this loads the data from disk

sort(suffix_array, fulltext)  # entries in the suffix_array are offsets into 
                              # the fulltext
dump(suffix_array)

It needs around 270MB, instead of over 400.



ObjectSpace + size - Eric Hodel (2006-12-03 (Sun) 17:20:50)

mem_inspect gives fairly accurate sizes including handling shared data in Array and String. It probably needs an audit for complete accuracy, though.

mfp 2006-12-04 (Mon) 12:57:44

yup, I remember that, I might send you a patch someday


Last modified:2006/12/02 03:58:03
Keyword(s):[blog] [ruby] [frontpage] [GC] [memory] [fragmentation]
References: