Context Navigation

← Previous Ticket
Next Ticket →

Ticket #48 (assigned defect)

Last modified 4 weeks ago

Hpricot Memory Leak Problem

Reported by:	thinkr	Owned by:	why
Priority:	blocker	Milestone:	0.5
Component:	lib/hpricot	Version:	0.4
Keywords:	memory leak	Cc:	rowan@…

Description

I use hpricot on my long-living daemon program. I think some memory leak is in on the hpricot. As time goes by, program's memory usage is up and up, increasingly. Here's my just sample test code showing this problem.

url = "http://blabla..."
def do_hpricot(url)
  h = Hpricot(open(url)).search("div#entries") do
    entry.at("div.from")['href']
  end
end
loop do
  do_hpricot(url)
  sleep rand(1)
end

Or, Are there any other way to collect this??

Change History

in reply to: ↑ 1 Changed 18 months ago by bigsmoke

cc rowan@… added
priority changed from major to blocker

Replying to why:

What platform? I don't get such behavior on Linux. Hpricot doesn't keep any C structures, so chances of a memory leak are very small.

Well, the problem could very well be in your use of JQuery. Anyway, I've bumped into a very severe memory-leak too. The following script won't win any beauty prices but it will show a significant leak on our Debian server:

#!/usr/bin/ruby

require 'rubygems'
require 'mechanize'

agent = WWW::Mechanize.new

$BASE_URL = "http://www.world-airport-codes.com"

# Loop 'a' through 'z'
('a'..'z').each do |letter|
  country_url = "#{$BASE_URL}/alphabetical/country-name/#{letter}.html"
  warn "GETting #{country_url} ..."
  country_page = agent.get(country_url)
  warn "Parsing #{country_url} ..."

  (country_page/"tr.one, tr.two").each do |tr|
    airport_url = $BASE_URL + tr.at("td[2]/a")["href"]
    warn "GETting #{airport_url} ..."
    airport_page = agent.get(airport_url)
    airport_columns = airport_page.at("div.columns")

    airport_code = airport_columns.at("div.column1 span.detail:nth-of-type(0)").inner_text.sub(/^:\s*([^\s]*).*$/, '\1')
    airport_name = airport_columns.at("div.column1 span.detail:nth-of-type(1)").inner_text.sub(/^:\s*([^\s]*).*$/, '\1')
    city_name = airport_columns.at("div.column1 span.detail:nth-of-type(4)").inner_text.sub(/^:\s*([^\s]*).*$/, '\1')
    country_name = airport_columns.at("div.column1 span.detail:nth-of-type(5)").inner_text.sub(/^:\s*([^\s]*).*$/, '\1')
    country_code = airport_columns.at("div.column1 span.detail:nth-of-type(6)").inner_text.sub(/^:\s*([^\s]*).*$/, '\1')
    long = airport_columns.at("div.column2 span.detail:nth-of-type(0)").inner_text.sub(/: (\d+)&#176; (\d+)&#8217; (\d+)&#8221; (W|E).*/, '\1 \2 \3 \4')
    lat = airport_columns.at("div.column2 span.detail:nth-of-type(1)").inner_text.sub(/: (\d+)&#176; (\d+)&#8217; (\d+)&#8221; (N|S).*/, '\1 \2 \3 \4')
    world_area_code = airport_columns.at("div.column2 span.detail:nth-of-type(2)").inner_text.sub(/^:\s*([^\s]*).*$/, '\1')

    unless long =~ /Unknown/ or lat =~ /Unknown/
      puts "#{airport_code};#{airport_name};#{city_name};#{country_name};#{country_code};#{long};#{lat};#{world_area_code}"

Oh, and please be gentile if you're testing this code on the www.world-airport-codes.com server. ;-) They're nice people.

Changed 17 months ago by badyacorp

Changed 7 months ago by rynok

I've seen memory growth as well, but I believe that it has something to do with references hanging around. Using this code to dump memory usage:

      # trying to see where our memory is going
      population = Hash.new{|h,k| h[k] = [0,0]}
      array_sizes = Hash.new{|h,k| h[k] = 0}
      ObjectSpace.each_object do |object|
        # rough estimates, see http://eigenclass.org/hiki.rb?ruby+space+overhead
        size = case object
               when Array
                 array_sizes[object.size / 10] += 1
                 case object.size
                 when 0..16
                   20 + 64
                 else
                   20 + 4 * object.size * 1.5
                 end
               when Hash;    40 + 4 * [object.size / 5, 11].max + 16 * object.size
               when String;  30 + object.size
               else 120 # the iv_tbl, etc
               end
        count, tsize = population[object.class]
        population[object.class] = [count + 1, tsize + size]

I see these results:

Array                  104898    8905770
String                 124861    7271214
Hpricot::ETag           18622    2234640
Hpricot::STag           18622    2234640
Hpricot::Elem           18622    2234640
Hash                    18896    1771080
Hpricot::Text           13924    1670880
Proc                     8608    1032960
Float                    7787     934440
Class                     382      45840
Gem::Version              295      35400

Which well after anything directly referencing the Hpricot::Doc has dropped out of scope (even tried wrapping in begin ... end). I've even added several GC.start's to no avail.

I'm wondering if there's some circular reference, but I'm not sure how I'd track that down.

Note: See TracTickets for help on using tickets.

Download in other formats:

Context Navigation

Ticket #48 (assigned defect)

Hpricot Memory Leak Problem

Description

Change History

follow-up: ↓ 2 Changed 21 months ago by why

in reply to: ↑ 1 Changed 18 months ago by bigsmoke

Changed 17 months ago by badyacorp

Changed 17 months ago by lwu

Changed 17 months ago by why

Changed 7 months ago by rynok

Changed 6 months ago by karevn

in reply to: ↑ description Changed 5 months ago by dacort

Changed 3 months ago by madcrock

Changed 4 weeks ago by tmm1

Download in other formats: