Multi-Vendor Based Data Network Topologies Testing At Scale
- Auto Generated Scaled Test Report on Camphor Networks Platform, Begin Time: 2022-01-09 01:15:48 +0000, End Time: 2022-01-09 01:55:17 +0000
- Infrastructure: Camphor Kubernetes Platform with 2 node(s)
- Number of Projects: 10, Number of test iterations per project: 10
- Number of Routers in each topology: Random[3, 10], Number of network connections in each topology: Random[9, 19]
- VNF Vendors: CRPD, MX, SRX, VRR, IOS, IOSXR, NXOS, EOS, FRR
- Total Duration using Traditional Serial Execution
- Time taken to boot up each VNF (VM) on average: 15 Minutes
- Average number of VNF VMs per test: 6
- Time taken to boot all VNFs (VMs) (serially) within a test: 6*15m == 90 Minutes
- Total time to configure each VNF and then for OSPF and BGP to converge: 10 Minutes
- Total time to run one test: 100 Minutes
- Total time required to run all 100 tests: 7 Days!
- Average Time Per Test: 100 Minutes
- Tests have to be run serially (Not trivial to run parallely, especially in a distributed elastic cluster of computes) (without K8!)
- Total Duration using Camphor Networks Platform
- Total number of reusable VNF VMs in each project: 10
- Total initial bring up time of 100 (recycled) VNFs (VMs) across 10 projects (in parallel): 1 Hour
- Total time to run 100 Tests: 1 Hour(One Time Bringup) + 40 Minutes(Tests Execution Time)
- Average Time Per Test: 1 Minute
- Total Tests: 100, Passed: 91, Failed: 9, Success Rate: 91%
- Net Improvement in Speed/Productivity/Cost: 100 Times
- All failures are due to some VNF VMs mis-behaving intermittently when configuration is committed
In these tests, following events are performed.
- Multiple projects are created in camphor networks platform
- Within each project, multiple tests (iterations) are run serially, one after another
- All projects run concurrently, thanks to horizontal scalability provided seamlessly by Camphor Kubernetes Platform
- Within each test (iteration) in each project, following steps are executed
- Random network topology is generated, with routers availed from different networking vendors
- Each node is configured to use OSPF as IGP
- node1 and node2 are designated as BGP Route Reflector
- Rest of the nodes are configured as BGP Route Reflector clients to both node1 and node2 route reflector server
- Entire configuration is committed
- Each configured node is instantiated. If a ready router (VM) is already available in free a pool, it is used instead of creating a new one afresh!
- All nodes’ interfaces are networked properly as dictated by the topology by Camphor Networks Platform automatically and automagically!
- All nodes establish OSPF neighbors with their neighbors thus ensuring full connectivity from any node to any other node in the topology
- All route reflector client nodes establish BGP peerings with both node1 and node2 route reflector servers
- Verification is done to ensure that all BGP sessions are indeed ESTABLISHED
- All devices and networks are deleted (Router VMs go back to free pool for future re-use!)
- Verification is done to ensure that cleanup is successfully complete
| Test | Status | Project | Iteration | Topology | Nodes | Edges | Duration |
|---|---|---|---|---|---|---|---|
| Test1 | PASS | project1 | 1 |  |
10 | 19 | 5 Minutes and 16 Seconds |
| Test2 | PASS | project1 | 2 |  |
9 | 18 | 3 Minutes and 42 Seconds |
| Test3 | PASS | project1 | 3 |  |
4 | 6 | 2 Minutes and 50 Seconds |
| Test4 | PASS | project1 | 4 |  |
7 | 13 | 3 Minutes and 46 Seconds |
| Test5 | PASS | project1 | 5 |  |
8 | 10 | 4 Minutes and 8 Seconds |
| Test6 | PASS | project1 | 6 |  |
3 | 3 | 2 Minutes and 17 Seconds |
| Test7 | PASS | project1 | 7 |  |
5 | 10 | 3 Minutes and 7 Seconds |
| Test8 | PASS | project1 | 8 |  |
10 | 10 | 4 Minutes and 26 Seconds |
| Test9 | PASS | project1 | 9 |  |
4 | 6 | 1 Minute and 28 Seconds |
| Test10 | PASS | project1 | 10 |  |
5 | 7 | 3 Minutes and 18 Seconds |
| Test11 | PASS | project2 | 1 |  |
10 | 19 | 4 Minutes and 51 Seconds |
| Test12 | PASS | project2 | 2 |  |
4 | 6 | 3 Minutes and 27 Seconds |
| Test13 | PASS | project2 | 3 |  |
10 | 19 | 5 Minutes and 28 Seconds |
| Test14 | PASS | project2 | 4 |  |
3 | 3 | 2 Minutes and 1 Second |
| Test15 | PASS | project2 | 5 |  |
6 | 8 | 4 Minutes |
| Test16 | PASS | project2 | 6 |  |
6 | 6 | 3 Minutes and 4 Seconds |
| Test17 | PASS | project2 | 7 |  |
9 | 13 | 3 Minutes and 58 Seconds |
| Test18 | PASS | project2 | 8 |  |
7 | 14 | 1 Minute and 19 Seconds |
| Test19 | PASS | project2 | 9 |  |
8 | 15 | 2 Minutes and 36 Seconds |
| Test20 | PASS | project2 | 10 |  |
8 | 9 | 3 Minutes and 43 Seconds |
| Test21 | PASS | project3 | 1 |  |
10 | 19 | 3 Minutes and 35 Seconds |
| Test22 | PASS | project3 | 2 |  |
7 | 16 | 4 Minutes and 34 Seconds |
| Test23 | PASS | project3 | 3 |  |
7 | 14 | 3 Minutes and 18 Seconds |
| Test24 | PASS | project3 | 4 |  |
9 | 18 | 3 Minutes and 25 Seconds |
| Test25 | PASS | project3 | 5 |  |
10 | 15 | 4 Minutes and 48 Seconds |
| Test26 | PASS | project3 | 6 |  |
3 | 3 | 2 Minutes and 27 Seconds |
| Test27 | PASS | project3 | 7 | |
3 | 3 | 1 Minute and 4 Seconds |
| Test28 | PASS | project3 | 8 |  |
7 | 13 | 3 Minutes and 34 Seconds |
| Test29 | PASS | project3 | 9 |  |
6 | 11 | 4 Minutes and 55 Seconds |
| Test30 | PASS | project3 | 10 |  |
10 | 13 | 3 Minutes and 16 Seconds |
| Test31 | PASS | project4 | 1 |  |
10 | 19 | 3 Minutes and 52 Seconds |
| Test32 | PASS | project4 | 2 |  |
5 | 10 | 4 Minutes and 27 Seconds |
| Test33 | PASS | project4 | 3 |  |
9 | 13 | 3 Minutes and 53 Seconds |
| Test34 | PASS | project4 | 4 |  |
3 | 3 | 2 Minutes and 47 Seconds |
| Test35 | PASS | project4 | 5 |  |
5 | 4 | 2 Minutes and 49 Seconds |
| Test36 | PASS | project4 | 6 |  |
4 | 3 | 3 Minutes and 5 Seconds |
| Test37 | PASS | project4 | 7 |  |
5 | 6 | 2 Minutes and 44 Seconds |
| Test38 | FAIL node1 Peers | project4 | 8 |  |
9 | 8 | 5 Minutes and 16 Seconds |
| Test39 | PASS | project4 | 9 |  |
4 | 6 | 3 Minutes and 34 Seconds |
| Test40 | PASS | project4 | 10 |  |
6 | 15 | 2 Minutes and 52 Seconds |
| Test41 | PASS | project5 | 1 |  |
10 | 19 | 4 Minutes and 22 Seconds |
| Test42 | PASS | project5 | 2 |  |
4 | 6 | 4 Minutes and 3 Seconds |
| Test43 | PASS | project5 | 3 |  |
10 | 9 | 4 Minutes |
| Test44 | PASS | project5 | 4 |  |
6 | 14 | 3 Minutes and 46 Seconds |
| Test45 | FAIL node1 Peers | project5 | 5 |  |
8 | 10 | 5 Minutes and 3 Seconds |
| Test46 | PASS | project5 | 6 |  |
10 | 9 | 3 Minutes and 37 Seconds |
| Test47 | PASS | project5 | 7 |  |
8 | 7 | 4 Minutes and 11 Seconds |
| Test48 | PASS | project5 | 8 |  |
9 | 8 | 4 Minutes and 55 Seconds |
| Test49 | PASS | project5 | 9 |  |
4 | 6 | 2 Minutes and 41 Seconds |
| Test50 | PASS | project5 | 10 |  |
3 | 3 | 2 Minutes and 7 Seconds |
| Test51 | PASS | project6 | 1 |  |
10 | 19 | 4 Minutes and 6 Seconds |
| Test52 | PASS | project6 | 2 |  |
9 | 16 | 3 Minutes and 56 Seconds |
| Test53 | PASS | project6 | 3 |  |
10 | 15 | 3 Minutes and 34 Seconds |
| Test54 | PASS | project6 | 4 |  |
5 | 10 | 3 Minutes and 27 Seconds |
| Test55 | PASS | project6 | 5 |  |
5 | 7 | 3 Minutes and 26 Seconds |
| Test56 | PASS | project6 | 6 |  |
9 | 14 | 5 Minutes and 6 Seconds |
| Test57 | PASS | project6 | 7 |  |
6 | 5 | 4 Minutes and 42 Seconds |
| Test58 | PASS | project6 | 8 |  |
3 | 3 | 3 Minutes and 18 Seconds |
| Test59 | PASS | project6 | 9 |  |
9 | 10 | 4 Minutes and 24 Seconds |
| Test60 | PASS | project6 | 10 |  |
3 | 3 | 2 Minutes and 36 Seconds |
| Test61 | PASS | project7 | 1 |  |
10 | 19 | 4 Minutes and 13 Seconds |
| Test62 | PASS | project7 | 2 |  |
8 | 13 | 3 Minutes and 42 Seconds |
| Test63 | PASS | project7 | 3 |  |
10 | 10 | 4 Minutes and 13 Seconds |
| Test64 | PASS | project7 | 4 |  |
9 | 18 | 3 Minutes and 20 Seconds |
| Test65 | PASS | project7 | 5 |  |
10 | 15 | 4 Minutes and 34 Seconds |
| Test66 | PASS | project7 | 6 |  |
6 | 15 | 3 Minutes and 20 Seconds |
| Test67 | FAIL node1 Peers | project7 | 7 |