Showing posts with label url. Show all posts
Showing posts with label url. Show all posts

Saturday, June 15, 2013

Ruby - keep track of urls you've already visited

How do I keep track or urls that I've already visit in my ruby projects? Here's what I do:

def visited? url
  @visited ||= []
  return true if @visited.include? url
  @visited << url
  false
end

Now I've got a visited? method that tells me if I've called it already on an url

visited?('http://www.google.com')
#=> false
visited?('http://www.google.com')
#=> true

That makes it easy to just do something like:

scrape(url) unless visited?(url)
This was a short post but this is something I always put in my ruby code and I'll be referencing it later.

Thursday, December 6, 2012

Convert relative urls to absolute in php with Phpuri

Here's another common problem, how do I convert my relative urls to absolute urls in php? In most scripting languages there's some built in class that can do this for you. Unfortunately php is really a web development language so general purpose libraries can be lacking.

I tested two popular solutions against my test case and in the end decided to create a simpllified 'port' of ruby's URI class. Let's take a look at the competition:
  • rel2abs - The nicest thing I can say about this solution is that it's the fastest. Unfortunately it failed 30% of my tests.
    Usage: rel2abs($rel, $base)
  • UrlToAbsolute - This one did fairly well, passing 90% of my tests. Keep in mind that many of the tests are rare edge cases, so I imagine real world success would be close to 100%. I could almost be happy with this one. Unfortunately the global namespace clutter it creates is a potential disaster, I decided it's best to steer clear of this one as well.
    Usage: url_to_absolute($base, $rel)
  • Phpuri - While it passed 100% of the tests, I will concede that the deck was stacked. In other words I had the tests in mind while writing it and the goal was specifically to pass those tests.
    Usage: phpUri::parse($base)->join($rel)

The Verdict:

Obviously I'm biased but I'm scoring this one for Phpuri.
Leave a comment if you disagree, I want to hear about it.

The Code:

require 'phpuri.php';
echo phpUri::parse('https://www.google.com/')->join('foo');
//==> https://www.google.com/foo

The Download:

View the project or download just the latest source