How to find the coordinates of the buttons on a canvas, and click on them after using Java and Selenium?

前端未结

关注

 2  1155

無奈伤痛 2020-12-04 03:37

I have an online calculator that i want to automate some operations for, like subtraction, division, etc. but the thing is that there are no elements since it is a canvas ap

2条回答

慢半拍i (楼主)

2020-12-04 04:11

The element is within an </code>. So to invoke <code>click()</code> on the elements within the <code><canvas></code> you have to:</p> <ul> <li>Induce <em>WebDriverWait</em> for the desired <em>frame to be available and switch to it</em>.</li> <li>Induce <em>WebDriverWait</em> for the desired <em>element to be clickable</em>.</li> <li><p>You can use the following solution:</p> <ul> <li><p>Code Block:</p> <pre><code>driver.get("https://www.online-calculator.com/full-screen-calculator/") new WebDriverWait(driver, 10).until(ExpectedConditions.frameToBeAvailableAndSwitchToIt(By.id("fullframe"))); WebElement canvas = new WebDriverWait(driver, 10).until(ExpectedConditions.elementToBeClickable(By.id("canvas"))); </code></pre></li> </ul></li> </ul> <hr> <h2>HTML5 Canvas</h2> <p>The element is only a container for graphics and is a rectangular area on an HTML page. By default, a canvas has no border and no content. However, an <code>id</code> attribute (to be referred to in a script), a <code>width</code> and <code>height</code> attribute are specified to define the size of the canvas. To add a border, the style attribute is used. An example:</p> <pre><code><canvas id="myCanvas" width="200" height="100" style="border:1px solid #000000;"></canvas> </code></pre> <p>The HTML canvas is a two-dimensional grid. The upper-left corner of the canvas has the coordinates (0,0).</p> <p>In the article Automated Testing of HTML5 Canvas Applications with Selenium WebDriver @Aaron Mulder mentions, interacting with the elements within <code><canvas></code> is possible using <em>event support</em> of the Actions <em>Class</em> API:</p> <ul> <li><p>moveToElement(WebElement target, int xOffset, int yOffset): Moves the mouse to an offset from the top-left corner of the element.</p> <pre><code>new Actions(driver).moveToElement(canvas, xWithinCanvas, yWithinCanvas).click().perform(); </code></pre></li> </ul> <p>But is not 100% reliable as, every <code>mouse down</code>, <code>mouse up</code>, or <code>mouse click</code> happens at the <strong>center</strong> of the element. So the code above produces a <code>mouse move</code> event to the provided coordinates (<strong><code>x</code></strong>,<strong><code>y</code></strong>), then a <code>mouse move</code> event to the center of the <code><canvas></code>, then a <code>mouse down</code>, <code>mouse up</code>, and <code>click</code> at the center of the <code><canvas></code>. That should have been fine for a <code><button></code> but is not worth for a <code><canvas></code>, where you want to be able to click at a specific location.</p> <p>The workaround, is to dispatch synthesized mouse events using JavaScript as follows:</p> <pre><code>// pageX and pageY are offsets which you need to know through mouse coordinates. ((JavascriptExecutor)driver).executeScript("var evt = $.Event('click', { pageX: " + x + ", pageY: " + (y + 55) + " } );" + "$('#myCanvas').trigger(evt);"); </code></pre> <p>However, to click on the elements within the <code><canvas></code> you can be at ease using firefox as the <code>mouse move</code> event works well in Firefox and you can avoid using the <em>mouse coordinates</em> as the event processing as follows:</p> <pre><code>new Actions(driver).moveToElement( canvas, xWithinCanvas, yWithinCanvas).perform(); ((JavascriptExecutor)driver).executeScript("$('#canvas').click();"); </code></pre> <hr> <h2>This usecase</h2> <p>To automate a <strong>substruction</strong> operation e.g. <kbd>3</kbd><kbd>-</kbd><kbd>1</kbd><kbd>=</kbd> using Selenium you can use the following solution:</p> <ul> <li><p>Code Block:</p> <pre><code>driver.get("https://www.online-calculator.com/full-screen-calculator/"); new WebDriverWait(driver, 10).until(ExpectedConditions.frameToBeAvailableAndSwitchToIt(By.id("fullframe"))); WebElement canvas = new WebDriverWait(driver, 10).until(ExpectedConditions.elementToBeClickable(By.id("canvas"))); //clicking on 3 new Actions(driver).moveToElement(canvas, 0, 0).moveByOffset(0,(255/6)*3).click().build().perform(); //clicking on the substract sign (-) new Actions(driver).moveToElement(canvas, 0, 0).moveByOffset((174/5)*2,(255/6)*3).click().build().perform(); //clicking on 1 new Actions(driver).moveToElement(canvas, 0, 0).moveByOffset(-(174/5)*4,(255/6)*3).click().build().perform(); //clicking on equals to sign (=) new Actions(driver).moveToElement(canvas, 0, 0).moveByOffset((174/5)*4,(255/6)*4).click().build().perform(); </code></pre></li> <li><p>Execution Video:</p></li> </ul> <p><img src="https://i.stack.imgur.com/NDJcU.gif" alt="AutomatingCanvasSelenium"></p> <hr> <h2>Reference</h2> <p>You can find a couple of relevant detailed discussion in:</p> <ul> <li>How to perform Mouse Wheel scrolling over HTML5 Canvas in Selenium?</li> </ul> </p> <div class="appendcontent"> </div> </div> <div class="jieda-reply"> <span class="jieda-zan button_agree" type="zan" data-id='783968'> <i class="iconfont icon-zan"></i> <em>0</em> </span> <span type="reply" class="showpinglun" data-id="783968"> <i class="iconfont icon-svgmoban53"></i> 讨论(0) </span> <div class="jieda-admin"> </div> <div class="noreplaytext bb"> <center><div> <a href="https://www.e-learn.cn/qa/q-225417.html"> 查看其它2个回答 </a> </div></center> </div> </div> <div class="comments-mod " style="display: none; float:none;padding-top:10px;" id="comment_783968"> <div class="areabox clearfix"> <form class="layui-form" action=""> <div class="layui-form-item"> <label class="layui-form-label" style="padding-left:0px;width:60px;">发布评论:</label> <div class="layui-input-block" style="margin-left:90px;"> <input type="text" placeholder="不少于5个字" AUTOCOMPLETE="off" class="comment-input layui-input" name="content" /> <input type='hidden' value='0' name='replyauthor' /> </div> <div class="mar-t10"><span class="fr layui-btn layui-btn-sm addhuidapinglun" data-id="783968">提交评论 </span></div> </div> </form> </div> <hr> <ul class="my-comments-list nav"> <li class="loading"> <img src='https://www.e-learn.cn/qa/static/css/default/loading.gif' align='absmiddle' />  加载中... </li> </ul> </div> </li> </ul> <div class="layui-form layui-form-pane"> <form id="huidaform" name="answerForm" method="post"> <div class="layui-form-item layui-form-text"> <a name="comment"></a> <div class="layui-input-block"> <script type="text/javascript" src="https://www.e-learn.cn/qa/static/js/neweditor/ueditor.config.js"></script> <script type="text/javascript" src="https://www.e-learn.cn/qa/static/js/neweditor/ueditor.all.js"></script> <script type="text/plain" id="editor" name="content" style="width:100%;height:200px;"></script> <script type="text/javascript"> var isueditor=1; var editor = UE.getEditor('editor',{ //这里可以选择自己需要的工具按钮名称,此处仅选择如下五个 toolbars:[['source','fullscreen', '|', 'undo', 'redo', '|', 'bold', 'italic', 'underline', 'fontborder', 'strikethrough', 'removeformat', 'formatmatch', 'autotypeset', 'blockquote', 'pasteplain', '|', 'forecolor', 'backcolor', 'insertorderedlist', 'insertunorderedlist', 'selectall', 'cleardoc', '|', 'rowspacingtop', 'rowspacingbottom', 'lineheight', '|', 'customstyle', 'paragraph', 'fontfamily', 'fontsize', '|', 'indent', '|', 'justifyleft', 'justifycenter', 'justifyright', 'justifyjustify', '|', 'link', 'unlink', 'anchor', '|', 'simpleupload', 'insertimage', 'scrawl', 'insertvideo', 'attachment', 'map', 'insertcode', '|', 'horizontal', '|', 'preview', 'searchreplace', 'drafts']], initialContent:'', //关闭字数统计 wordCount:false, zIndex:2, //关闭elementPath elementPathEnabled:false, //默认的编辑区域高度 initialFrameHeight:250 //更多其他参数，请参考ueditor.config.js中的配置项 //更多其他参数，请参考ueditor.config.js中的配置项 }); editor.ready(function() { editor.setDisabled(); }); $("#editor").find("*").css("max-width","362px"); </script> </div> </div> <div class="layui-form-item"> <label for="L_vercode" class="layui-form-label">验证码</label> <div class="layui-input-inline"> <input type="text" id="code" name="code" value="" required lay-verify="required" placeholder="图片验证码" autocomplete="off" class="layui-input"> </div> <div class="layui-form-mid"> <span style="color: #c00;"><img class="hand" src="https://www.e-learn.cn/qa/user/code.html" onclick="javascript:updatecode();" id="verifycode"><a class="changecode" href="javascript:updatecode();"> 看不清?</a></span> </div> </div> <div class="layui-form-item"> <input type="hidden" value="225417" id="ans_qid" name="qid"> <input type="hidden" id="tokenkey" name="tokenkey" value=''/> <input type="hidden" value="How to find the coordinates of the buttons on a canvas, and click on them after using Java and Selenium?" id="ans_title" name="title"> <div class="layui-btn layui-btn-disabled" id="ajaxsubmitasnwer" >提交回复</div> </div> </form> </div> </div> <input type="hidden" value="225417" id="adopt_qid" name="qid" /> <input type="hidden" id="adopt_answer" value="0" name="aid" /> </div> <div class="layui-col-md4">  <dl class="fly-panel fly-list-one"> <dt class="fly-panel-title">热议问题</dt>