Skip to main content

Drop in application throughput on AWS (Mystery Unresolved)

I have posted a question on stackoverflow about sudden reduction in application throughout. I also follows up on this conversation with AWS support and Following is the gist of conversation with AWS Support -
tl dr:
AWS instances my team was using was not set with enhanced networking capabilities to be able to get max n/w performance (i.e. 10 gigabits on c4.8xlarge instance). For example ixgbevf on test-aws-am was not set to 2.14.2.
Long Version:
The test in question is a static html page. GET request, no complicated logic, No EBS, DJ-CTS etc.
  • What n/w capabilities should we experience when instance does not satisfy enhanced networking capabilities?
    AWS Support: We really do not have specific numbers because that varies... by a lot of factors (time of day, other instances sharing network in same location.. and several other factors... What we do, is to advice customers to do benchmarking tests to confirm that the instances meet the performance expectations... for the applications
  • During the test 18430 KiloBytes/sec data is transferred which is way under the limit of 10 gigabits n/w.
    AWS support has been insistent on probability of throughput throttle being an application issue.
    AWS Support: Through our testing we have eliminated ELB as a potential bottleneck for the drop in throughput and we know that the issue is occurring at the back-end instances. Looking at the back-end instances, the general performance metrics such as CPU utilization, Network In, Network Out etc for both the instances looks good. This indicates that the issue we are facing could be an application issue. could you please add another instance and try the test again?
    I can not confirm on this since there have been zero errors during multiple iterations of test runs. Once tests hit the lower limit on throughput then any subsequent test run shows the results in the lower range of throughout (about 8000 requests/sec) but if I wait for few hours (about 2 to 3 hrs) and run test then it is back to same behavior that is higher throughput for about half hour and then back to 8000 requests/sec. I excluded the possibilities of adding another instances as testing without ELB (described below) exhibits same behavior.
  • ELB Prewarm did not help on addressing reduced throughput. Moreover warm up is time bound and it scales down after some time. The warmup rather showed the more skewed results and throughput dropped in 5 minutes than without it - 



Before Warmup >>


After Warmup >>



  • Different instance type for test agent from m4.4xlarge to m4.10xlarge and c4.large have shown same behaviour of throttle in application throughput. Test runs with ramp up period of increasing threads hold same behavior barring that it reaches the higher limit after greater amount of time attributed to the ramp up period and not half an hour
  • Testing without ELB exhibits same behavior except that the high throughput would be in the range of 8000 requests/sec and then it drops to about 4000 requests/sec. Despite tests use custom DNS resolver; this further excludes any anomaly caused by probable DNS caching of ELBs.

Mystery of drop in application throughput was never solved.

Later on I got to know from in house n/w engineers that it is not only enhanced n/w capabilities but also placement group which is required to be able to use 10 gigabit n/w performances. Unfortunately placement group is limited to having instances in same availability zone. Is not it risky?

Popular posts from this blog

Appium and android mobile app automation

Next appium and Android mobile app automation video tutoria l is live. If you are new to appium then please check - appium-tutorial This video tutorial covers - Start vysor (Just for this session and not mobile automation :)) Start appium and start appium inspector Desired Capabilities platformName - Android deviceName - L2N0219828001013 (as seen on "adb devices") Saved Capability Sets Start Session Scan app elements using appium inspector Get appPackage and appActivity using "APK info" app Install "APK info" app and open app whose appPackage and appActivity are required i.e. calculator Check top section of app icon com.android.calculator2 is app package com.android.calculator2.Calculator is app activity testng.xml file settings for running Android app tests Test details com.seleniumtests.tests.mobile.AndroidAppTest and CalculatorScreen class View beautiful STF test report  

Return only first or last element from webelements collection

Note: If you are new to java and selenium then start with selenium java training videos .     We often come across situation when there are multiple elements on a page and we probably like to exercise only a few of them using selenium webdriver. May be just first and last element. For example on a search result page we may like to click on only first and last link and not all. This is when Iterables API comes handy. (By the way I am assuming that you have already completed watching selenium training videos :)). Once we have collection of web element then we can use Iterables to get only first or last element as following - Consider that we fetch collection of element as - List< WebElement > webElements = getDriver().findElements(By. id ( "htmlID" ));   Now we can get the first web element from this collection as -  WebElement firstElement = Iterables. getFirst (webElements,  getDriver().findElement(By. id ( "defaultElement" )));   Herein second

FAQ

Back to Appium Tutorial Index Element Location Strategy  -  Launch UIAutomationViewer as -  uiautomatorviewer & Following is an example for UIAutomationViewer and using it to find elements on Android App. Herein an image of UIAutomationViewer with calculator application -  The Element 64 can be identified as -  driver.findElement(By.name("64")); driver.findElement(By.id("com.android.calculator2:id/formula")); driver.findElement(By.className("android.widget.EditText")); Element location by TagName is deprecated. Refer Stackoverflow  post1  and  post2  for more on this. You can also use xPath strategy to identify element as described in this  post . Appium also supports  Mobile JSON Wire Protocol  locator strategy - -ios uiautomation : a string corresponding to a recursive element search using the UIAutomation library (iOS-only) -android uiautomator : a string corresponding to a recursive element search