Is it possible for WebDriver to click an element with a mouseclick event that invokes a JavaScript file that includes a test tracker?

问题

This is a question I have that arises from another user's question. If you look at my answer there, you will get some context for this question. The URL for the web page I am going to is https://hotels.ctrip.com/hotel/347422.html?isFull=F#ctm_ref=hod_sr_lst_dl_n_1_8 if you want to check it out for yourself.

Consider the python selenium script found at the bottom of the question. Nothing happens when selenium tries to click on this element:

browser.find_element_by_xpath('//*[@id="cPageBtn"]').click()

Same thing with this element

browser.find_element_by_xpath('//*[@id="appd_wrap_close"]').click()

When debugging my selenium script for each element, I confirmed that selenium can find the element just fine; it is not hidden, inside an iFrame, disabled, or any other oddity that I normally check for failed selenium actions.

However, it has a mouseclick event that invokes an interesting JavaScript file, and I was actually able to access it by navigating to URL shown here:

Hovering over the URL revealed that it is https://webresource.c-ctrip.com/ResUnionOnline/R3/float/floating_normal.min.js?20190306:2.

At the very beginning of the file I found

document.getElementById("ab_testing_tracker") && "abTestValue_Value" != h ? 
document.getElementById("ab_testing_tracker").value

So I search (via CSS selector in the dev console) the webpage HTML for an element with an id of "ab_testing_tracker" and I am less than surprised that it returns nothing. Then I unminified and searched through the file for all instances of "ab_testing_tracker". That led me to this element:

document.getElementsByTagName("body")[0].insertAdjacentHTML("afterBegin","<input type='hidden' name='ab_testing_tracker' id='ab_testing_tracker' value='"+h.split("|")[1]+"'>")

Well, it appears there is a hidden input node inserted in the body of the document for the purpose of automation tracking. Google searching revealed that automation tracking is often accomplished by looking at the navigator.userAgent property and finding userAgents that indicate automation. But the script is using a random legitimate userAgent every time, so I don't think the userAgent is how the detection is finding selenium.

Summary and possible workarounds

Selenium can't click certain elements on the web page likely due to testing tracking by the website. There are a couple things I thought of to get around it: maybe I can disable click events when using selenium? This I don't know how to do and couldn't find a way after searching online. Next, I tried to click on it with a Javascript executor, but that didn't work.

Does anyone know a way to get around the test tracker and click the desired element?

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By

# url
url = "https://hotels.ctrip.com/hotel/347422.html?isFull=F#ctm_ref=hod_sr_lst_dl_n_1_8"

# User Agent
User_Agent_List = ["Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.2 (KHTML, like Gecko) Chrome/22.0.1216.0 Safari/537.2",
                   "Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)",
                   "Mozilla/5.0 (compatible; MSIE 10.0; Macintosh; Intel Mac OS X 10_7_3; Trident/6.0)",
                   "Opera/9.80 (X11; Linux i686; U; ru) Presto/2.8.131 Version/11.11",
                   "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.2 (KHTML, like Gecko) Chrome/22.0.1216.0 Safari/537.2",
                   "Mozilla/5.0 (Windows NT 6.2; Win64; x64; rv:16.0.1) Gecko/20121011 Firefox/16.0.1",
                   "Mozilla/5.0 (iPad; CPU OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5355d Safari/8536.25"]

# Define the related lists
Score = []
Travel_Types = []
Room_Types = []
Travel_Dates = []
Comments = []

DEFINE_PAGE = 10

def next_page():

    current_page = int(browser.find_element_by_css_selector('a.current').text)

    # First, clear the input box
    browser.find_element_by_id("cPageNum").clear()
    print('Clear the input page')

    # Second, input the next page
    nextPage = current_page + 1
    print('Next page ',nextPage)
    browser.find_element_by_id("cPageNum").send_keys(nextPage)

    # Third, press the goto button
    browser.find_element_by_xpath('//*[@id="cPageBtn"]').click()



def scrap_comments():
    """
    It is a function to scrap User comments, Score, Room types, Dates.
    """
    html = browser.page_source
    soup = BeautifulSoup(html, "lxml")
    scores_total = soup.find_all('span', attrs={"class":"n"})
    # We only want [0], [2], [4], ...
    travel_types = soup.find_all('span', attrs={"class":"type"})
    room_types = soup.find_all('a', attrs={"class":"room J_baseroom_link room_link"})
    travel_dates = soup.find_all('span', attrs={"class":"date"})
    comments = soup.find_all('div', attrs={"class":"J_commentDetail"})
    # Save score in the Score list
    for i in range(2,len(scores_total),2):
        Score.append(scores_total[i].string)
    Travel_Types.append(item.text for item in travel_types)
    Room_Types.append(item.text for item in room_types)
    Travel_Dates.append(item.text for item in travel_dates)
    Comments.append(item.text.replace('\n','') for item in comments)

if __name__ == '__main__':

    # Random choose a user-agent
    user_agent = random.choice(User_Agent_List)
    print('User-Agent: ', user_agent)

    # Browser options setting
    options = Options()
    options.add_argument(user_agent)
    options.add_argument("disable-infobars")

    # Open a Firefox browser
    browser = webdriver.Firefox(options=options)
    browser.get(url)


    browser.find_element_by_xpath('//*[@id="appd_wrap_close"]').click()

    page = 1    
    while page <= DEFINE_PAGE:
        scrap_comments()
        next_page()

    browser.close()

回答1:

The problem is not the tracking or the click event, the problem is timing and possibly browser size. Maximize the browser window and add explicit wait when searching for the banner close button

browser = webdriver.Firefox(options=options)
browser.maximize_window()
browser.get(url)

wait = WebDriverWait(browser, 10)

wait.until(EC.element_to_be_clickable((By.ID, 'appd_wrap_close'))).click()
wait.until(EC.invisibility_of_element_located((By.ID, 'appd_wrap_default')))

current_page = int(browser.find_element_by_css_selector('a.current').text)
next_page = current_page + 1

page_number_field = wait.until(EC.visibility_of_element_located((By.ID, 'cPageNum')))
page_number_field.clear()
page_number_field.send_keys(next_page)
wait.until(EC.element_to_be_clickable((By.ID, 'cPageBtn'))).click()

来源：https://stackoverflow.com/questions/55204297/is-it-possible-for-webdriver-to-click-an-element-with-a-mouseclick-event-that-in