Ok, so you ran a bunch of tests using TabJolt. What next?
You are probably here because you read the article on how to conduct scalability testing
The first thing that you should do is extract the data out of postgres rather than making live connections.
Then, you get confused with all the results that you see in the workbook. What are you supposed to make out of all those numbers?
If you have never done any test of this kind before, you are in for a surprise.
At Tableau, we like to keep things simple. But, unfortunately, a lot of you see those test results seem anything but simple.
Here’s my take on it.
See, typically in companies, most of this stuff is typically discussed in room full of techies. and business folk are just informed if the test worked or not.
(sort of reminds me of a hospital scene with the patient, i.e. our Tableau Server has some tests done on it and the doctor (IT) comes out and tells you if your server survived or crashed)
But I am surprised by so many of the Tableau community who are under taking these tests by themselves and they are a bit techie but still sitting on the business side of things.
This has been both rather encouraging and bothersome for me.
Encouraging because I am all for giving people the info they need and asking them with more tools. So, kudos to all of you who have never done a load test before but felt comfortable to do it with TabJolt.
Bothersome because I have talked to folks recently who I feel are now armed with this info but are trying to make sense of this. Problem is that some of these folks don’t have a clue.
this reminds me of an X Ray Technician interpreting the results of some tests.
Anyway, I digress.
Well, that’s part of the reason behind my latest series of posts. To educate you more about the process behind the tests.
If you follow the approach I have outlined in one of my previous posts, I believe you will be in a much better position to understand the results rather than just running some random tests.
So, Where were we?
oh yes, let’s talk a out the TabJolt terms.
This is the amount of time you wants the tests to run. If you enter 120, TabJolt will generate samples for 120 seconds
also known as thread count.
This is a simulation of an action On a Tableau Server that is either viewing a dashboard or interacting with it.
Also, keep in mind when TabJolt creates these threads, it is not using different logins or Tableau users.
They are all using the same user or guest account
simply put, this is a user just filtering the viz or clicking somewhere on it.
This is just the number of interactions generated by TabJolt.
So, if you refer to the image below for my first test, it means that TabJolt used a single thread and genrated 137 samples in 120 seconds. Since it is a single thread, it waited for the response before another interaction (or sample) was generated. Since on an average each interaction was taking .867 seconds, it was able to generate 137 samples.
Avg success response time
should be simple enough to understand. average time taken by each request to complete. This metric doesn’t not include any errored requests
Avg response time
similar to above but also includes errors
90 percentile response time
This is kind of important. This the time in msec that 90% of the requests completed within.
So, if you generated 1000 samples and this metric is 927 msec, it means that 900 (90% of 1000) requests completed in .927 seconds.
95 percentile response time
similar to above except this is for 95% requests.
Why is 90 and 95 PCT important?
Well, even though servers are machines and everything should be consistent, we all know sometimes some random things happen.
These eliminate the impact of the outliers. So, if you had those 1000 samples but for some reason about 10 of those request (1% of total) got stuck somewhere and succeeded in 75 seconds, the average would become 1.667 from 0.927.
So, I typically interpret using average but also look at 95 PCT.
Hope this helps understand a few things.
What are we trying to achieve as we ramp up the users?
It is to get the highest possible vusers on the system with AVG success TPS being as close to the AVG with a single user
In other words, looking for a tipping point like in the example below
Hope this helps, see you next time… when I will explain some of the weirdness in my test results