Menu Close

Network Topology Regression Tests

Multi-Vendor Based Data Network Topologies Testing At Scale

  • Auto Generated Scaled Test Report on Camphor Networks Platform, Begin Time: 2022-01-09 01:15:48 +0000, End Time: 2022-01-09 01:55:17 +0000
  • Infrastructure: Camphor Kubernetes Platform with 2 node(s)
  • Number of Projects: 10, Number of test iterations per project: 10
  • Number of Routers in each topology: Random[3, 10], Number of network connections in each topology: Random[9, 19]
  • VNF Vendors: CRPD, MX, SRX, VRR, IOS, IOSXR, NXOS, EOS, FRR

  • Total Duration using Traditional Serial Execution
    • Time taken to boot up each VNF (VM) on average: 15 Minutes
    • Average number of VNF VMs per test: 6
    • Time taken to boot all VNFs (VMs) (serially) within a test: 6*15m == 90 Minutes
    • Total time to configure each VNF and then for OSPF and BGP to converge: 10 Minutes
    • Total time to run one test: 100 Minutes
    • Total time required to run all 100 tests: 7 Days!
    • Average Time Per Test: 100 Minutes
    • Tests have to be run serially (Not trivial to run parallely, especially in a distributed elastic cluster of computes) (without K8!)

  • Total Duration using Camphor Networks Platform
    • Total number of reusable VNF VMs in each project: 10
    • Total initial bring up time of 100 (recycled) VNFs (VMs) across 10 projects (in parallel): 1 Hour
    • Total time to run 100 Tests: 1 Hour(One Time Bringup) + 40 Minutes(Tests Execution Time)
    • Average Time Per Test: 1 Minute

  • Total Tests: 100, Passed: 91, Failed: 9, Success Rate: 91%
  • Net Improvement in Speed/Productivity/Cost: 100 Times

  • All failures are due to some VNF VMs mis-behaving intermittently when configuration is committed

In these tests, following events are performed.

  1. Multiple projects are created in camphor networks platform
  2. Within each project, multiple tests (iterations) are run serially, one after another
  3. All projects run concurrently, thanks to horizontal scalability provided seamlessly by Camphor Kubernetes Platform
  4. Within each test (iteration) in each project, following steps are executed
    1. Random network topology is generated, with routers availed from different networking vendors
    2. Each node is configured to use OSPF as IGP
    3. node1 and node2 are designated as BGP Route Reflector
    4. Rest of the nodes are configured as BGP Route Reflector clients to both node1 and node2 route reflector server
    5. Entire configuration is committed
    6. Each configured node is instantiated. If a ready router (VM) is already available in free a pool, it is used instead of creating a new one afresh!
    7. All nodes’ interfaces are networked properly as dictated by the topology by Camphor Networks Platform automatically and automagically!
    8. All nodes establish OSPF neighbors with their neighbors thus ensuring full connectivity from any node to any other node in the topology
    9. All route reflector client nodes establish BGP peerings with both node1 and node2 route reflector servers
    10. Verification is done to ensure that all BGP sessions are indeed ESTABLISHED
    11. All devices and networks are deleted (Router VMs go back to free pool for future re-use!)
    12. Verification is done to ensure that cleanup is successfully complete

Test Status Project Iteration Topology Nodes Edges Duration
Test1 PASS project1 1 10 19 5 Minutes and 16 Seconds
Test2 PASS project1 2 9 18 3 Minutes and 42 Seconds
Test3 PASS project1 3 4 6 2 Minutes and 50 Seconds
Test4 PASS project1 4 7 13 3 Minutes and 46 Seconds
Test5 PASS project1 5 8 10 4 Minutes and 8 Seconds
Test6 PASS project1 6 3 3 2 Minutes and 17 Seconds
Test7 PASS project1 7 5 10 3 Minutes and 7 Seconds
Test8 PASS project1 8 10 10 4 Minutes and 26 Seconds
Test9 PASS project1 9 4 6 1 Minute and 28 Seconds
Test10 PASS project1 10 5 7 3 Minutes and 18 Seconds
Test11 PASS project2 1 10 19 4 Minutes and 51 Seconds
Test12 PASS project2 2 4 6 3 Minutes and 27 Seconds
Test13 PASS project2 3 10 19 5 Minutes and 28 Seconds
Test14 PASS project2 4 3 3 2 Minutes and 1 Second
Test15 PASS project2 5 6 8 4 Minutes
Test16 PASS project2 6 6 6 3 Minutes and 4 Seconds
Test17 PASS project2 7 9 13 3 Minutes and 58 Seconds
Test18 PASS project2 8 7 14 1 Minute and 19 Seconds
Test19 PASS project2 9 8 15 2 Minutes and 36 Seconds
Test20 PASS project2 10 8 9 3 Minutes and 43 Seconds
Test21 PASS project3 1 10 19 3 Minutes and 35 Seconds
Test22 PASS project3 2 7 16 4 Minutes and 34 Seconds
Test23 PASS project3 3 7 14 3 Minutes and 18 Seconds
Test24 PASS project3 4 9 18 3 Minutes and 25 Seconds
Test25 PASS project3 5 10 15 4 Minutes and 48 Seconds
Test26 PASS project3 6 3 3 2 Minutes and 27 Seconds
Test27 PASS project3 7 3 3 1 Minute and 4 Seconds
Test28 PASS project3 8 7 13 3 Minutes and 34 Seconds
Test29 PASS project3 9 6 11 4 Minutes and 55 Seconds
Test30 PASS project3 10 10 13 3 Minutes and 16 Seconds
Test31 PASS project4 1 10 19 3 Minutes and 52 Seconds
Test32 PASS project4 2 5 10 4 Minutes and 27 Seconds
Test33 PASS project4 3 9 13 3 Minutes and 53 Seconds
Test34 PASS project4 4 3 3 2 Minutes and 47 Seconds
Test35 PASS project4 5 5 4 2 Minutes and 49 Seconds
Test36 PASS project4 6 4 3 3 Minutes and 5 Seconds
Test37 PASS project4 7 5 6 2 Minutes and 44 Seconds
Test38 FAIL node1 Peers project4 8 9 8 5 Minutes and 16 Seconds
Test39 PASS project4 9 4 6 3 Minutes and 34 Seconds
Test40 PASS project4 10 6 15 2 Minutes and 52 Seconds
Test41 PASS project5 1 10 19 4 Minutes and 22 Seconds
Test42 PASS project5 2 4 6 4 Minutes and 3 Seconds
Test43 PASS project5 3 10 9 4 Minutes
Test44 PASS project5 4 6 14 3 Minutes and 46 Seconds
Test45 FAIL node1 Peers project5 5 8 10 5 Minutes and 3 Seconds
Test46 PASS project5 6 10 9 3 Minutes and 37 Seconds
Test47 PASS project5 7 8 7 4 Minutes and 11 Seconds
Test48 PASS project5 8 9 8 4 Minutes and 55 Seconds
Test49 PASS project5 9 4 6 2 Minutes and 41 Seconds
Test50 PASS project5 10 3 3 2 Minutes and 7 Seconds
Test51 PASS project6 1 10 19 4 Minutes and 6 Seconds
Test52 PASS project6 2 9 16 3 Minutes and 56 Seconds
Test53 PASS project6 3 10 15 3 Minutes and 34 Seconds
Test54 PASS project6 4 5 10 3 Minutes and 27 Seconds
Test55 PASS project6 5 5 7 3 Minutes and 26 Seconds
Test56 PASS project6 6 9 14 5 Minutes and 6 Seconds
Test57 PASS project6 7 6 5 4 Minutes and 42 Seconds
Test58 PASS project6 8 3 3 3 Minutes and 18 Seconds
Test59 PASS project6 9 9 10 4 Minutes and 24 Seconds
Test60 PASS project6 10 3 3 2 Minutes and 36 Seconds
Test61 PASS project7 1 10 19 4 Minutes and 13 Seconds
Test62 PASS project7 2 8 13 3 Minutes and 42 Seconds
Test63 PASS project7 3 10 10 4 Minutes and 13 Seconds
Test64 PASS project7 4 9 18 3 Minutes and 20 Seconds
Test65 PASS project7 5 10 15 4 Minutes and 34 Seconds
Test66 PASS project7 6 6 15 3 Minutes and 20 Seconds
Test67 FAIL node1 Peers project7 7 9 9 4 Minutes and 56 Seconds
Test68 PASS project7 8 7 12 3 Minutes and 46 Seconds
Test69 FAIL node1 Peers project7 9 8 9 4 Minutes and 40 Seconds
Test70 PASS project7 10 3 3 1 Minute and 44 Seconds
Test71 PASS project8 1 10 19 3 Minutes and 17 Seconds
Test72 FAIL node1 Peers project8 2 7 13 6 Minutes and 20 Seconds
Test73 PASS project8 3 7 6 3 Minutes and 11 Seconds
Test74 PASS project8 4 4 4 4 Minutes and 24 Seconds
Test75 PASS project8 5 9 17 4 Minutes and 6 Seconds
Test76 PASS project8 6 10 15 8 Minutes and 13 Seconds
Test77 PASS project8 7 10 17 47 Seconds
Test78 PASS project8 8 3 2 1 Minute and 52 Seconds
Test79 PASS project8 9 5 7 2 Minutes and 53 Seconds
Test80 PASS project8 10 9 16 2 Minutes and 55 Seconds
Test81 PASS project9 1 10 19 3 Minutes and 27 Seconds
Test82 PASS project9 2 3 3 4 Minutes
Test83 PASS project9 3 3 2 3 Minutes and 13 Seconds
Test84 FAIL node1 Peers project9 4 9 8 4 Minutes and 34 Seconds
Test85 PASS project9 5 9 12 3 Minutes and 23 Seconds
Test86 PASS project9 6 10 11 4 Minutes and 23 Seconds
Test87 PASS project9 7 7 16 3 Minutes and 32 Seconds
Test88 PASS project9 8 10 11 4 Minutes and 33 Seconds
Test89 PASS project9 9 10 10 4 Minutes and 13 Seconds
Test90 PASS project9 10 10 18 1 Minute and 2 Seconds
Test91 FAIL node1 Peers project10 1 10 19 5 Minutes and 2 Seconds
Test92 PASS project10 2 3 3 3 Minutes and 18 Seconds
Test93 FAIL node1 Peers project10 3 8 14 4 Minutes and 57 Seconds
Test94 PASS project10 4 9 17 2 Minutes and 40 Seconds
Test95 PASS project10 5 3 3 2 Minutes and 24 Seconds
Test96 PASS project10 6 8 10 3 Minutes and 48 Seconds
Test97 PASS project10 7 8 14 1 Minute and 1 Second
Test98 PASS project10 8 9 18 3 Minutes and 45 Seconds
Test99 PASS project10 9 7 12 3 Minutes and 27 Seconds
Test100 PASS project10 10 7 8 51 Seconds