Scraping data from another website?

[Mick]

Anyone know how to scrape data from another website?

Reason i ask is, the owner of the website does not have an API or XML available.


What im trying to accomplish is, using a search button with I POST GET POST deal to my .php page.


I searched .NET but all is confusing.

Any ideas?

Mick.

[IchBin]

Without an xml or some type of rss feed, you are going to be hacking your way at the code. You'll have to create a script that will grab the page, and then parse that page for certain elements which you can then grab and put into your page or something. It really is not something I would advise to do. If the person changes their page, then your script doesn't work.

[Mick]

Without an xml or some type of rss feed, you are going to be hacking your way at the code. You'll have to create a script that will grab the page, and then parse that page for certain elements which you can then grab and put into your page or something. It really is not something I would advise to do. If the person changes their page, then your script doesn't work.

I understand that if the owner changes the page,my script wont work.   The owner of the other website doesnt have the know how and neither do i.

I tried the iFrames tags, but quite frankly it sucks and i dont want the header and footer of the other website.   I just want to use the content of that certain page.

[JPDeni]

Did you try copying the source code and pasting it into an html article? Take out the stuff you don't want and leave the stuff you do want.

[Mick]

Actually i tried that but it didnt work as i copied the source for the search bar.   But the return didnt work me.


Heres the deal,    i play RockBand, im sure you heard over it.   This website www.gorockband.com uses a search bar where anyone can input their player name and view their scores and progression.  In this case my gamer ID is:

fastnfuriouz

If you input my name, it will show all my scores and ranks.

In short, i would like to add that "search bar" in my website ( i have the source code) and use the results in an article page or custom action.

Im assuming to use the scores, i would do a FORM POST, connect to the site (via HTTP) with a search value and i would get the HTML as a result.

Next i would have to parse the file to extract the scores and display them the way i want.


Does this make any sense?

[JPDeni]

I don't know if what you want to do is possible. At least I wouldn't know how to do it.

[mc]

If you can't directly access the code for the search bar, then you'll need to look into CURL to access the other site, and Regular Expressions (RegEx) to filter the page for displaying the search bar on it's own.

Although the other hack you could try, is to see if the search would work by using the GET method, instead of POST, that way you can submit a search directly to the other site using your own search form, and then it's just a case of filtering the returned page for the results.
To find out if that's possible, you'd need to look at the code for the search form, then try manually building an url with the form fields submitted as GET values in the url.

Whether it'll work, will depend on if the page handling the search request, has been written using $_POST or $_REQUEST array

[Mick]

This is the search bar source code:

Code Select Expand

<form accept-charset="UTF-8" method="post" action="http://www.gorockband.com/search/en" name="search_main" onsubmit="javascript: searchSubmit()">
<!-- Hack for IE.
WHY: Hitting ENTER on the form does not work.  Adding this invisible input fixes it. I also use it to get submitsearch to work in all browsers. -->
<input name="submithidden" value="Search" type="hidden">

<input name="currentpage" value="" type="hidden">
<input name="showcount" value="" type="hidden">
<input name="sortorder" value="" type="hidden">
<input name="scoressortorder" value="" type="hidden">

<table style="border: 1px solid rgb(102, 102, 102); background-color: rgb(51, 51, 51);" cellpadding="2" cellspacing="0" width="100%">
<tbody><tr>
<td><span style="color: darkorange;">Search</span></td>
<td><!-- onchange="javascript: changeFilter(this.options[this.selectedIndex].value);" -->
<select name="search_type" id="search_type">
<option>Leaderboards</option>
<option>Songs</option>

<!-- <option >Band mates</option> -->
</select></td>
<td width="100%"><input class="searchscoresbox" name="searchtext" value="" maxlength="255" type="text"></td>
<td><input style="margin: 0px 2px 0px 5px;" value="Search" name="submitsearch" type="submit"></td>
</tr>
</tbody></table>



<input name="song_source" id="song_source" value="" type="hidden">
<input name="solo_band_song" id="solo_band_song" value="" type="hidden">
<input name="song_difficulty" id="song_difficulty" value="" type="hidden">



</form>

This what it looks like in my website:   http://www.rocklive-dlc.com/index.php/page,12.html

add my name   fastnfuriouz and youll see my scores.

The URL in the code is their results page.   

[mc]

You'll also need to look for the javascript function called searchSubmit() that is called when the form is submitted.

[Mick]

I'll give it a whirl tonight after work.  I would think I also need the values I want for the return.