The visual editor in Wordpress 2.8 is fucked for the time being (javascript errors), at least if you use anything besides IE from what I understand.
So hold off on it, was going to do a post today, but now my mojo is gone, maybe later.
                |
This post is a re-write. I didn’t think the last version was very good, was too long, and looks like it even went out of date. This version is shorter, up to date (as of right now), and easier to follow I think.
For the sake of teaching, I’m going to pick a fairly easy target. We are going to write a pretty simple Google scraper. A Google scraper is just complex enough to get your dick wet, while at the same time posing a few easy to solve problems. We are not interested in scraping all the information from the page we are going to focus on some general information: URL, Title, Description, and Cite.
In these beginning stages I tend to write things down, usually just in something like notepad. Things like format of URLs, where certain information your trying to grab, any other problems you may run into.
The rest of this article is just going to be my observations and notes. I’ll be using Firebug to pull the page apart.
The general Google url looks something like this:
http://www.google.com/search?q=huge+fish&hl=en&start=10&sa=N
Variables:
I used the search term huge fish because it gives us all the types of results that are possible, video, image, and search.
Page Information (Click To Enlarge)
To make things easier I colored and numbered the code we are looking at. Red boxes and numbers are the actuall ad. Green boxes and numbers are the corosponding code.
You’ll also notice two light blue boxes around the <li class=”g“> and the <div class=”s“>. These are both importantant to notice as they represent an actuall search entry (not video or image). Both needed to determain if it is indeed a search entry. We need to make note of this when building the scraper.
The only other thing we haven’t located was the URL. It’s in the line above the green #1 box:
<a class=”l“ onmousedown=”return clk(this.href,”,”,’res’,'3′,”)“ href=”http://fishosaur.com/“ realurl=”http://fishosaur.com/“>
We’ll just pull it out of that href field.
That ends part 3. We know where all our data is, now we just need to get down to coding. Next up, the backbone of a scraper, cURL.
| |
This story broke yesterday on WF by Jon:
BREAKING NEWS!
This is not a rumor. This is 100% confirmed.
Today there were as many as 4-5 subpeonas served to AFFILIATES who promoted Acai Berry offers and are based in the state of Illinois.
UPDATED 06/04/09: All of the affiliates that were served with subpeonas are Illinois based, and were allegedly using Oprah in their ads, on Facebook only. All were shown information that could have only been attained via a subpeona of Facebook. The Illinois attorney general subpeona has requests for financial information, campaign information, tracking information, networks and advertisers tied to the campaigns, etc. So they are obviously following the typical AG investigating protocol. Still strange enough to see them going after affiliates first and then working their way up, as its usually the opposite. But this is shaping up to become a big mess as I’ve been told by other state AG’s that they are very interested in following this investigation and very likely will begin their own or a possible joint investigation (lets hope it doesn’t get to that). The affiliates involved have asked to remain anonymous, so please keep people’s names to yourselves if you can, and respect their privacy wishes for a change. Bottom line, this is being fueled by Oprah/her lawyers, Illinois AG, Facebook + FTC partnership.
There have been rumors from the Harpo camp close to the Facebook/Acai advertiser subpeonas that Myspace/Newscorp was to be subpeonad for advertiser account information too, but this has not been confirmed or commented on by anyone from Newscorp or the Illinois AG as of yet.
For privacy reasons I will not name any of the affiliates who were subpeonad today, but this came as a pretty big shock as the Illinois Attorney General had some pretty in-depth knowledge of the people involved, more so than usual.
UPDATED 06/04/09: I still strongly believe there is an industry insider ratting, but its my opinion and I’ll just keep it to myself now as I’m not out to destroy anyone’s reputation without 100% definitive proof. Lets see how it plays out unfortunately.
This is the first time that a state attorney general has gone after ONLY the largest affiliates, one by one, as if they were reading it off of a numbered hit list.
Before you even ask, yes, most of the guys on the list are members here. If they want to come out and speak about it, that’s up to them, so please don’t turn this into a rumor mill or witch hunt. I’ve got most of their names confirmed and confirmation from an anonymous source with the Illinois Attorney Generals office that this is in fact quite real.
The only good side of it is that they don’t seem to be seeking out any jail time, just civil action. I can absolutely assure you that this is not going to be the last time we see any of this happen, as there is an unconfirmed suit by another state AG finishing up and getting ready to go after a handful of advertisers and networks, but no affiliates that I know of.
UPDATED 06/04/09: Boutique affiliate/agency Bloosky was hit with a suit from the AG in Utah on Tuesday over their Google bizopp (whatever the name is, they all sound the same), so not Acai related, but still interesting, will make a seperate thread about this later. It doesn’t look like anything very serious either and will most likely be settled out of court quickly. I know for a fact that there is no mention of affiliates in this at all, it seems to be more about customer complaints who couldn’t figure out how to cancel/refund (typical, they know how to file with the AG but can’t figure out how to call for a refund).
| |
Yesterday I had the pleasure to talk to Yousif Yalda, who is the owner of Profit Kings Media, a brand new network. Yousif Yalda is a successful affiliate marketer himself, so you can expect that he knows what the typical affiliate goes through, and how he can make a better network. If you hang out of WickedFire his screen name is GrindHard.
Enough of my yapping, lets get to it:
MADPPC: First Question, how did you get into affiliate/internet marketing?
PKMYousif: First of all we would like to thank you for taking the time to talk to us. I got started into affiliate marketing rather by surprise. I was browsing a forum one day and I stumbled upon a guy who responded to my thread — and the topic wasn’t even relevant. I hit him up one day to thank him for giving me such a valuable response and we got into talking about what we both do for a living. At the moment, I was attending school full-time and working on various web application security ventures. He explained to me how much he made, and I immediately thought to myself that he was specializing in stocks or some other industry I had no insight about. He later explained to me in brief what affiliate marketing was all about, and so I was interested and started doing my research. A few weeks later with some search and asking questions, I was ready to get started! Pretty random, it turned out really well!
MADPPC: Now I understand you have your own network now?
PKMYousif: Yes, I do. It’s called Profit Kings Media and you can check us out at http://www.ProfitKingsMedia.com ! We’re a brand new network and taking off really well. We’re changing up the industry one network at a time, and we believe that overtime we’ll be the solution that all will turn to for real affiliate marketing. We understand from experience the flaws that exist for such a long time with cash flow, greedy networks taking HUGE margins, and more — and the way we operate shows these solutions in plain sight.
MADPPC: Now from what you told me already, this network sounds like a great opportunity for anybody whether your a seasoned pro or just getting started in the game. Many people when they first start out can’t do the standard $1000 a week to get wires that most networks want. Can you tell us a little bit more about how you payout?
PKMYousif: Sure, there’s always been a struggle with cash flow since most networks can’t seem to offer a better payout. This is a business , and what business do you know pays it’s employees on a 30 day basis? Let’s get real for a second here. The entire CPA model is by default in the advertisers favor, and we know from experience evaluating leads does not take such a lengthy time. I understand the need for that cap as only ’serious affiliates’ should be allowed this sort of payout structure, but this is a disadvantage for any affiliate to get started and progress. We understand that you need cash flow to help with re-fueling and resuming investments with paid traffic or any other resources that are needed. With Net 30, you really limit the affiliates performance. This is our intake, it differs greatly with other networks. I won’t go into the business side of why most networks do this, as it has its benefits but we’re financially stable to do this and believe in it. We pay out weekly via Check, Wire Transfer, and ACH Deposit.
MADPPC: And that’s no matter how much you do in that week?
PKMYousif: Whether you’re making $30 or $300,000 – we will pay you weekly.
MADPPC: Now that’s something I have NEVER heard of a network doing. What tracking platform do you run on?
PKMYousif: Well, we plan on killing one network at a time, and this is just one of the many strategies we use to operate. I have a few friends who run their own networks and share the same visions as Profit Kings Media and the directions we are taking to move forward. We are using Linktrust as our tracking system.
MADPPC: About how many offers do you offer right now?
PKMYousif: We have over 200 offers under testing, and about 30 offers live. We only carry offers that are truly reporting high ROI and convert well. Many networks publicize hundreds of offers, but how many are truly worth promoting? We are adding new offers daily and you can expect to turn serious loot.
MADPPC: Those kind of networks I typically favored. Understandably your a new network, but how many affiliate managers do you have right now? Also, what kind of help can they give to enhance the affiliates experience, and most importantly, their bottom line?
PKMYousif: At the present time, I don’t feel the need of hiring anyone as I can handle the work load thus far. On a side note, I do have 2 upcoming affiliate managers who will join on board with PKM soon. PKM as a whole helps with bidding, keywords, LP’s, adcopys, images, traffic source sources, and more. We work over 16 hours a day, 7 day a week, serving you with guidance and advice that will help you make more money. We are affiliates ourselves and we know where to run offers and how.
MADPPC: One more thing I wanted to touch on before we wrap this up, lets say I have some sort of offer I’d like to promote on your network as the merchant. What are your requirements?
PKMYousif: We love exclusives, and work closely to provide full-blown management to help launch whatever product you have in mind. In fact, we have 3+ exclusives that have yet to hit the industry and have chosen us to be their primary partner in helping to generate the most revenue possible with accurate reporting in quality and volume. You have an idea, now let’s put it out there!
MADPPC: Lastly, for the sake of all the referral whores, do you have a referral system?
PKMYousif: Unfortunately, not at this time! With some extensive feedback, we figured that most affiliates like to keep 100% commission, and I believe that if you’re in this industry to earn an actual income and are serious, then you shouldn’t depend on referrals. That’s just one way to look at it. It’s a nice chunk of change, but at this time it’s not needed as this is the decision we’ve made based on opinions from affiliates we work with. In the future, we might consider offering a life-time referral program.
[End of Interview]
I wish Yousif Yalda and Profit Kings Media the best of luck. It seems like some of the best networks right now, where founded by affiliates (Ads4Dough, Max Bounty, Convert2Media) so hopefully we get another one. As Yousif had mentioned they test offers to see what converts the best and use them, out of the 20 or so offers he hits many of the more popular verticals: Dating (several offers), Crush, Education, Acai, Colon, Anti-Wrinkle, Google Cash, Insurance, Credit Reports.
Some of you will be bummed out with the no-referral, but in honestly you can’t get the best possible payout if someone has a referral on you. What you thought that 2-5% came out of the networks pocket? Fuck no, it comes out of the affiliates pocket.
Also another huge point plus for smaller affiliates is WEEKLY payouts – no matter how much you make.
BTW I get nothing for doing this interview, just in case some of you might think that. Check them out!
| |
Now we get the idea of POST and GET. We found our target, we know it’s url structure, we know where the data is, but how do we use PHP to fetch the webpages?
Luckily we have what is call cURL (from PHP.net):
PHP supports libcurl, a library created by Daniel Stenberg, that allows you to connect and communicate to many different types of servers with many different types of protocols. libcurl currently supports the http, https, ftp, gopher, telnet, dict, file, and ldap protocols. libcurl also supports HTTPS certificates, HTTP POST, HTTP PUT, FTP uploading (this can also be done with PHP’s ftp extension), HTTP form based upload, proxies, cookies, and user+password authentication.
Basically when it comes to interacting with the net, there’s not much cURL can’t do. I like to think about it as a completely controllable web browser.
Now chances are when you first start using cURL you’ll look at the manual on PHP.net and go bonkers. Where do I start? What should I know? Well, get ready for a cURL crash course. At the end of this will be a little cURL template that you can use to save you some time in future projects.
I like to break cURL up into a few little segments: Initialization/Closure, General Options, Cookies, GET/PUT, URL Execution and Retreival.
To initialize a cURL resource it’s as easy as:
$ch = curl_int();
$ch becomes our cURL resource. Why the variable $ch? Because that’s what the manual uses, and what I use. And to close a cURL resource:
curl_close($ch);
In between curl_int and curl_close there lots of options that can be set using curl_setop. A few I use on nearly every scraper are:
CURLOPT_SSL_VERIFYPEER – Used to tell cURL whether or not to verify a peers certificate. TRUE by default. I always use FALSE.
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
CURLOPT_RETURNTRANSFER - Very important. FALSE will have cURL just dump all the code onto the page, TRUE will allow us to capture all the info into a variable fo further processing.
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
CURLOPT_HEADER – TRUE to output the header. Doesn’t say if FALSE is the default value, but I always set it just because.
curl_setopt($ch, CURLOPT_HEADER, FALSE);
CURLOPT_USERAGENT – Your browsers User Agent. You don’t need this for all scrapers, but some sites throw a shit fit if you don’t include it. So as a good rule of thumb just include it. You can use Live HTTP Headers to tell what your current User Agent is, or just use the one I’m going to post.
$agent = 'User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008100922 Ubuntu/8.04 (hardy) Firefox/3.0.3'; curl_setopt($ch, CURLOPT_USERAGENT, $agent);
CURLOPT_FOLLOWLOCATION – TRUE to follow any “Location:” header. Basically to follow all redirects. Important, should always be TRUE.
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
Setting cookies is pretty easy:
CURLOPT_COOKIESESSION - From the cURL manual:
TRUE to mark this as a new cookie “session”. It will force libcurl to ignore all cookies it is about to load that are “session cookies” from the previous session. By default, libcurl always stores and loads all cookies, independent if they are session cookies or not. Session cookies are cookies without expiry date and they are meant to be alive and existing for this “session” only.
You almost always want this to be TRUE, on rare occasions depending on how the website acts, you may have to load the old session cookies.
curl_setopt($ch, CURLOPT_COOKIESESSION, TRUE);
CURLOPT_COOKIEFILE – A file to read/write all cookie data from/to.
$cookie = '/tmp/cookie.txt'; curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie);
CURLOPT_COOKIEJAR – Basically a file that cURL will write all internal cookies to when the connection closes.
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie);
Well I kinda lied, GET is the same as just setting a URL and variables manually, retrieving the page with no header modification. So we’ll actually cover GET in the next section. However Post is a little bit more complicated, and less common.
CURLOPT_POST – TRUE to do a POST.
curl_setopt($ch, CURLOPT_POST, TRUE);
CURLOPT_POSTFIELDS – From the cURL manual:
The full data to post in a HTTP “POST” operation. To post a file, prepend a filename with @ and use the full path. This can either be passed as a urlencoded string like ‘para1=val1¶2=val2&…‘ or as an array with the field name as key and field data as value. If value is an array, the Content-Type header will be set to multipart/form-data.
I find it easiest to just use the urlencoded string, although I’m sure some of you are fanboys for the arrays. You fanboys can get bent.
$post = 'var1=val1&var2=val2'; curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
CURLOPT_URL – The value of your URL.
$url = 'http://www.domain.tld/index.php?var1=val1&var2=val2'; curl_setopt($ch, CURLOPT_URL, $url);
curl_exec – Finally, we execute and capture our returning output in the $rawdata variable:
$rawdata = curl_exec($ch);
NOTE: One of the biggest errors I see other coders make is they think they have to reset all their options every time they want to execute cURL. In the same script, you only have to change the options you want to change between curl_exec. What you have set will stay set. So most the time all you have to do is just change the URL and run curl_exec again.
When you put it all together you’ll have a nice little cURL skeleton like this:
<?php $cookie = '/tmp/cookie.txt'; $agent = 'User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008100922 Ubuntu/8.04 (hardy) Firefox/3.0.3'; $post = 'var1=val1&var2=val2&var3=val3'; $url = 'http://www.domain.tld/index.php?var1=val1&var2=val2'; $ch = curl_init(); curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE); curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE); curl_setopt($ch, CURLOPT_HEADER, FALSE); curl_setopt($ch, CURLOPT_USERAGENT, $agent); curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE); //POST - don't include for GET. curl_setopt($ch, CURLOPT_POST, TRUE); curl_setopt($ch, CURLOPT_POSTFIELDS, $post); //cookies curl_setopt($ch, CURLOPT_COOKIESESSION, TRUE); curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie); curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie); curl_setopt($ch, CURLOPT_URL, $url); $rawdata = curl_exec($ch); curl_close($ch); ?>
Out.
| |
Understanding the fundamentals of how sites communicate with themselves, and how we communicate with them, is crucial in being able to reverse engineering a site for our scraper. Luckily it’s pretty easy for the most part.
Anatomy of a URL
Image 2.1
There really are only two ways how a site passes information: GET and POST.
There’s only minor differences between them. Image 2.1 is a demonstration of passing information using GET variables. GET variables you’ll see right in your address bar, and have a fixed size limit. POST variables are passed in the header, are wrote the same way as GET variables, are not directly seen by the user, have no fixed size limit, and looks much cleaner. POST variables are slightly “safer” but pretty much just as easy to read.
Live HTTP Headers Example 1: GET
In this first example, I went to yellowpages.com. In Find I put: chinese. In the Location box I put: chicago. Pull up Live HTTP Headers, make sure “Capture” is clicked, and click “Find” on the yellowpages.com page.
Image 2.2 (click to enlarge)
You should get a similar response. The bold numbers indicate:
The URL:
http://www.yellowpages.com/search?search_terms=chinese&geo_location_terms=chicago&x=21&y=11
You’ll also notice that our information we typed in are in the search_terms variable (chinese), and the geo_location_terms variable (chicago). I have bolded them for illustration purposes.
There’s lots of other stuff there, but chances are you won’t have to use any of it. That’s more of an advance subject that I won’t be covering in this series.
Live HTTP Headers Example 2: POST
POST is not used as commonly as GET are – if you wrote code you’d understand why. Usually you see POST being used for logins. So I headed over to Facebook and just tried to login with some fake credentials. Email: aguy@someone.com Password: apssword.
Image 2.3 (click to enlarge)
You should get back something similar. The bold numbers indicate:
This particular string of variables is pretty long, there’s some extra stuff in their that Facebook is passing itself, but the whole thing looks like this:
charset_test=%E2%82%AC%2C%C2%B4%2C%E2%82%AC%2C%C2%B4%2C%E6%B0%B4%2C%D0%94%2C%D0%84&locale=en_US&email=aguy%40someone.com&pass=apssword&pass_placeholder=&charset_test=%E2%82%AC%2C%C2%B4%2C%E2%82%AC%2C%C2%B4%2C%E6%B0%B4%2C%D0%94%2C%D0%84
You’ll see our email is in the email variable (aguy%40someone.com) %40 is really just the @ symbol. Certain characters in a URL have to be encoded, and you’ll find a nice guide outlining all the encodings here. You’ll also see our password in the pass variable (apssword). I have bolded them for illustration purposes.
Fin
That’ll end Part 2. You should have a pretty good idea of how URLs are encoded, variables are passed, and the basic usage of the Live HTTP Headers plugin for Firefox. Stay tuned, we are just starting.
| |
Welcome to the Noobies Guide to Scraping: Part 1. In this installment we are only going to focus on a few very basic things that we are going to need to get started, and no code will be wrote.
What is scraping? Scraping is the process of getting / gathering data from some web source, whether it’s off an rss feed or web-page or some other source. A program that automates the scraping (data collection) function is called a “Scraper“.
Still a little confused? Lets use a website as an example, like yellowpages.com. Yellowpages.com has tons of useful data you may want like addresses, phone numbers, business names. Let’s say for example, you wanted to build a small directory of business names, phone numbers, and addresses. Chances are it would take you ages to collect all the info you want, so why not build a scraper that gathers all the info and automates the process?
Have you ever heard the adage “Work smarter, not harder?”. There’s no point to waste your valuable time when some simple software can do it for you. Before we begin there’s a few things your going to need:
Firefox. That’s right, the mother of all browsers. Not just because the other tools we will use require it but because it’s a much better browser than IE.
Firebug. Every web developer should have this plugin in their arsenal. It is a swiss army knife of web-development. You may not need it for basic scrapers, but it’s better to have than not.
Live HTTP Headers. An integral part of seeing what kind of information being passed while using a website. Very useful even for basic scrapers, I find myself using it a ton.
Our scraper will be built in PHP utilizing cURL, as as much as this is a noobie tutorial this isn’t a lesson in PHP. Your expected to know at least basic-intermediate PHP.
| |
This is NOT an April fools joke guys. A severe SQL injection vulnerability was discovered in Prosper202 today. Wes and team have already patched and released an updated version. From Wes and crew:
This is an urgent message from the prosper202 team. Recently, we’ve found a security hole in our software. More Details of the security hole will be posted soon. As of now the Prosper202 team has fixed the problem and patched the security hole, so we encourage all users to download prosper version 1.2.1.
You can download version 1.2.1 Here
If you are running Prosper202 you need to UPDATE now! There have been many servers breached already. The list is growing by the minute. The WickedFire crew has already found out who was doing many break ins. We know his name, company, e-mail, phone, address, affiliate network id’s, the whole fucking works. However their still trying to get more info and find out who has been broken into. If you have been broken into please go to the link posted below and let the WF crew know.
You can follow all the action in this thread.
| |
So you have a brand new shiny site and your eager to compete in the SERPS. There’s just one problem, your not in the SERPS. Nor do you have any idea on how to get into the SERPS, and you wanted to be in the SERPS like yesterday.
Luckily it’s not that hard, and I’d consider it more of an art than anything because the things I will discuss are not wrote in stone. Sometimes it’s all in how you put it together. These tips, if executed correctly should typically get you indexed in 1-4 days.
I’m also going to use wordpress sites as an example, because wordpress sites have some natural advantages to them.
Factor: Proper SEO
Wordpress already does lots of great SEO things. One thing to look out for is to make sure your urls are proper search engine friendly. Under Settings -> Permalinks -> Click “Custom Structure” and in the text box put “/%postname%/” without quotes.
Plugin recommendation: All in One SEO Pack.
Factor: Content
Google loves content. Don’t have any? What reason does Google have to add you to the index when the Google Bot finally comes around?
Content, even fake made up spun content, can work. Start with some content and be prepared to continually ad it during the process of trying to get indexed.
Factor: Site Map
Google bot loves to see the site map so that it can easily navigate around. You’ll want a plugin that automatically generates and submits your site maps. After you have it set up make sure you run a web bot crawl test, like this one over at SEOmoz. It’s not my favorite ever, but the one I use to use all the time seems to have gone off line. If you have a good one give me a shout out.
Anyways, you want to run a web bot crawl test to make sure that the bot has permission to actually crawl around the site. I can’t tell you how many times I set something wrong in a header or on a sitemap and then the Google bot can’t get to my content, which means it has no reason to index me. From my experience it takes considerably longer to get indexed if you fuck it up the first time around like that. Learn from my mistake.
Recommended Plugin: Google XML Sitemaps
Factor: Ping List
A large comprehensive ping list will help start spreading your site faster than butter melts in a microwave. As you post content you always want it to get spread as far as it can, this can only help.
I’ve posted my ping list here.
Factor: Backlinks
Probably the most important thing to do. A good social bookmaking package will get in you in no time, but most people don’t have same day turn around.
What you need to look for are do-follow links. The difference between follow links and no-follow links are kind of obvious, one a bot can follow, on they aren’t suppose to.
The best advice I can give to you is to go find a very busy website, that has do-follow links, and just keep whoring yourself out. Google bot will soon be at your site to check it out.
Best of luck to you guys, you can do it!
| |
