JavaScript Scrapy and Javascript scrapy javascript pagination,scrapy javascript support,scrapy javascript click,scrapy

I'm trying to scrap some information from this ASP page http://laredoute.fr/ppdp/prod-350007615.aspx mainly the first 4 images in high resolution that load in the image-carousel. Depending on the color of the product you have some buttons that you must select. This is the code I have now that retrieves the buttons and adds them to a list to be clicked on later.

from selenium import webdriver
from scrapy.http import HtmlResponse
from scrapy.spiders import Spider
from scrapy.selector import Selector
from scrapy.selector import HtmlXPathSelector
from scrapy.linkextractors import LinkExtractor
import urllib
import urllib2
from bs4 import BeautifulSoup



class MyOpener(urllib.FancyURLopener):
    version = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.57 Safari/537.17'

myopener = MyOpener()
url = 'http://www.laredoute.fr/ppdp/prod-350007615.aspx'

f = myopener.open(url)
soup = BeautifulSoup(f)

viewstate = soup.find("fieldset", class_="set set-colour")


list = []

for elem in viewstate.findAll('label'):
    #print elem['title']
    #print elem['for']
    list.append(elem['for'].strip('\n\t ,'))

#print list


driver = webdriver.Chrome('/Users/vasquez/Documents/crawler/chromedriver')

driver.maximize_window()
driver.get(url)

radio = driver.find_element_by_id(list[0])


driver.execute_script("arguments[0].click();", radio)

Now the part I'm having problems with is this one. That image carousel is here in this part of HTML code.

<div class="divProds jcarousel-clip">

    <ul class="divAddScroller">

    </ul>

</div>

If I open the Developer Tools in Chrome and click over it the whole code appears, if I parse the whole HTML with Scrapy as I done before the code is not there and I can retrieve the img link that I need. This is the part of the HTML that I want to parse.

<li><a target='_blank' href="javascript:void(0)">

<img src="//media.laredoute.com/products1/72by72/d/e/6/350007615_0_PR_1_11970785_350007615-1fca06aa-305f-4b3f-92da-80e8e21cb43a_1200.jpg" data-src="http://media.laredoute.com/|Dimension|/d/e/6/350007615_0_PR_1_11970785_350007615-1fca06aa-305f-4b3f-92da-80e8e21cb43a_1200.jpg" title="Blouse manches longues, transparences, dentelle VERO MODA" alt="Blouse manches longues, transparences, dentelle VERO MODA image 1" width="72" height="72" data-cerberus="img_pdp_thumbnails1" class="">
</a>
</li>

So as a final question, how do I make Scrapy load that javascript:void(0) img link down below. Thanks.

Answer:1

I was working on a small JavaScript project to make a tile based game (not homework, don't worry!), however I ran into a problem with my 'tile' class. Each 'tile' has an X and a Y property, for its X ...

I was working on a small JavaScript project to make a tile based game (not homework, don't worry!), however I ran into a problem with my 'tile' class. Each 'tile' has an X and a Y property, for its X ...

I'm using Ramda as my functional programming helper library to build a React app. I'm trying to build myself whereAny. Ramda exposes where which checks if every prop it is given satisfies the ...

I'm using Ramda as my functional programming helper library to build a React app. I'm trying to build myself whereAny. Ramda exposes where which checks if every prop it is given satisfies the ...

I have a form on a PHP page : <form name="SendEmail01" method="post"> form items here... </form> I then process it like this: if(isset($_POST["Submit"])) { All tasks here... $...

I have a form on a PHP page : <form name="SendEmail01" method="post"> form items here... </form> I then process it like this: if(isset($_POST["Submit"])) { All tasks here... $...

the problem is simple, document.getElementsByTagName('*') does not select the SVG tag and in the console it gives an error. But if I erase the SVG labels, works correctly. My Code: var Master = ...

the problem is simple, document.getElementsByTagName('*') does not select the SVG tag and in the console it gives an error. But if I erase the SVG labels, works correctly. My Code: var Master = ...

  1. document getelementsbytagname java
  2. document getelementsbytagname innerhtml
  3. document getelementsbytagname w3schools
  4. document getelementsbytagname multiple tags
  5. document getelementsbytagname radio button
  6. document.getelementsbytagname('video') 0 .request picture in picture()
  7. document.getelementsbytagname(name)
  8. document.getelementsbytagname is not a function
  9. document.getelementsbytagname('body')
  10. document.getelementsbytagnamens
  11. document.getelementsbytagname(...).addeventlistener is not a function
  12. document.getelementsbytagname( input )
  13. document.getelementsbytagname foreach
  14. document.getelementsbytagname(...).appendchild is not a function
  15. document.getelementsbytagname( body ) 0
  16. document.getelementsbytagname('head') 0 .appendchild(script)
  17. document.getelementsbytagname jquery
  18. document.getelementsbytagname js
  19. document.getelementsbytagname('base') 0 .href
  20. document.getelementsbytagname('script') 0