Htykuut

What I need: Count the number of reviews under an extension in Chrome Store in all languages.
What I did: Tried BeautifulSoup to extract a certain tag. I reserched the html-code of the page and found a review tag:

Tried this code:

    from bs4 import BeautifulSoup

    import requests

    page = requests.get(url)

    soup = BeautifulSoup(page.content, 'html5lib')

    comments = soup.find_all('div', class_ = 'ba-bc-Xb ba-ua-zl-Xb')

But print(comments) shows that the array is empty.

I am stuck at the moment and I see that further I need to handle two problems:

1. How to cope with select language buttom? How to count reviews in all languages if by default only one language is selected.
   




2. The reviews are stored in different tabs. I read about dynamically extract it but didn't get a point.

You could use selenium to perform the tasks and waits for page changes and extract the review count from the PaginationMessage. Tested with a few links. You may need to add error handling for items with no reviews. There also seems to be some POST XHR activity yielding review JSON strings that you may wish to explore.

from selenium import webdriver

from selenium.webdriver.support.wait import WebDriverWait

from selenium.webdriver.support import expected_conditions as EC

from selenium.webdriver.common.by import By

from selenium.webdriver.common.action_chains import ActionChains



url = 'https://chrome.google.com/webstore/detail/evernote-web-clipper/pioclpoplcdbaefihamjohnefbikjilc?hl=en/'

#url = 'https://chrome.google.com/webstore/detail/https-everywhere/gcbommkclmclpchllfjekcdonpmejbdp?hl=en/'

d = webdriver.Chrome()

d.get(url)

WebDriverWait(d, 5).until(EC.visibility_of_element_located((By.ID, ':21'))).click()

ActionChains(d).click_and_hold(WebDriverWait(d, 5).until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.h-z-Ba-ca.ga-dd-Va.g-aa-ca')))).perform()

languageSelection = WebDriverWait(d, 5).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, '.g-aa-ca-ma-x-L')))

languageSelection[1].click()



s= WebDriverWait(d, 5).until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.Aa.dc-tf + span'))).text

print(s.split()[-1])

d.quit()

try this

from selenium import webdriver

from selenium.webdriver.common.by import By

from selenium.webdriver.support.ui import WebDriverWait 

from selenium.webdriver.support import expected_conditions as EC 

import time



driver = webdriver.Chrome()



driver.get('https://chrome.google.com/webstore/detail/evernote-web-clipper/pioclpoplcdbaefihamjohnefbikjilc?hl=en')

wait = WebDriverWait(driver, 5)

wait.until(EC.visibility_of_element_located((By.ID, ':21'))).click()

wait.until(

        EC.visibility_of_element_located((By.CSS_SELECTOR, '.h-z-Ba-ca.ga-dd-Va.g-aa-ca'))

    ).click()



english = driver.find_element_by_xpath('//div[@class="ah-mg-j"]/span').text

print('English: ' + english.split()[-1])



wait.until(

        EC.visibility_of_element_located((By.XPATH, '//div[@class="g-aa-ca-ma-x-L" and text() = "All languages"]'))

    ).click()

wait.until_not(EC.text_to_be_present_in_element((By.XPATH, '//div[@class="ah-mg-j"]/span'), english))

time.sleep(2)



AllCount = driver.find_element_by_xpath('//div[@class="ah-mg-j"]/span').text

print('All languages: ' + AllCount.split()[-1])

driver.close()