Over the past few months, I have had the privilege of working with the SCinet WAN team to design and build the network that would support this year’s Supercomputing conference (SC18) in Dallas, TX. SCinet, dubbed “the world’s fastest temporary network” delivered 4.02 terabits per second of capacity to the exhibition hall, allowing vendors and research institutions to demonstrate the latest and greatest in supercomputing/HPC technology and applications.
The 2018 SCinet network, comprised of equipment, software, and services from 40 different vendors (with a value of around $52M) was designed over the course of 9 months, built and tested in one week, and then wheeled a quarter of a mile through the convention center halls the next week, to be operational for less than a week! Talk about a whirlwind of events!
CALIENT’s involvement in SCinet (as with the majority of our applications) revolved around automating the fiber patching, primarily for circuit testing and turn-up. In previous years, as is typical with most deployments, testing required engineers to fiber a test set to the appropriate port (after cleaning and scoping) and setting up a test for a variable duration (30 minutes to 36 hours), finally leaving the setup until the tests were completed. If the test failed, the technician would need to identify the problem, which may be local or hundreds of miles away in a carrier’s facility, and then either address the problem locally, or submit a ticket for remote problem resolution.
While waiting for the issue to be resolved so testing could resume on that port, engineers might want to use the test set on another port, which would involve moving the test set, more cleaning, more scoping, etc. You can see the tedium and waste of multiple human, network, and test resources involved here. Adding to the mix at SCinet is the fact that all of this was being done during the one week of staging, while members of the Routing, Fiber, Power and DevOps teams were also jockeying for position around the racks to work on their equipment.
If only there were a way to do this remotely and without the challenges and delays of re-fibering! And there is! This is one of the applications where optical circuit switches deliver tremendous value. A CALIENT S320 switch was installed in the SCinet WAN transport racks and fibered between all the client-side ports on the Ciena, Infinera, and Juniper optical transport systems, and internal network core routers from Cisco and Juniper. (See SCinet diagram below)
Several test sets (generously donated by Viavi) were also connected to the S320 via a 12-strand spool of fiber from our work table. With this setup, we could sit back and remotely cross connect the test sets to circuits that were ready to be tested. In this scenario, fibers were scoped and cleaned once (never to be moved again), and if a test failed for any reason, a few keystrokes was all that was needed to move the test set and start testing a different circuit. “Set It and Forget It” as Ron Popeil would say! Not only did this create a much more efficient testing procedure, it also meant that once a circuit was tested and passed, we didn’t have to risk the possible negative consequences of removing the test set and re-fibering.
After a week of staging was complete, the whole SCinet team headed home for a week to rest up for the coming storm… moving the entire system to the show floor! This “down week” is also used for teams to work out any kinks that were not able to be addressed during staging. For the WAN team this typically means finishing up circuits that had issues on the far end. Since circuit testing usually didn’t get started until later in the week, this only left a few days for testing while the team was on site.
Activity during the down week involved a lot of coordination with carriers and data centers to track down issues along the long-haul circuits. We would have a carrier install a soft or hard loop at some point in the circuit so that we could verify some portion of it while waiting for the rest of the work to be completed. Typically, this would require a technician onsite at the SCinet staging site to move the test sets around. Not this year however! The OCS allowed us to work remotely and utilize any of the available Viavi, Ixia, or Spirent test ports on any circuit we chose to. (Thanks to Spirent who generously loaned us a 12 port dX3 card with 12 100Gig Quint Speed optics!)
“The addition of the CALIENT switch to the SCinet architecture gave us tools we didn’t have in the past. It allowed for quick remote testing, and the ability to shift circuits and traffic automatically among other benefits.”
Hans Addleman, Deputy Co-Chair, SCinet WAN Team
Speaking of the Ixia ports, they really highlighted another great OCS use case. The Ixia ports technically belonged to the Routing team, and 100Gig ports are something you usually don’t want to loan out as you never know when you will get them back. With the OCS in place, borrowing an optic was as simple as a few key strokes! If Routing wasn’t using Ixia ports for a few hours (or minutes), we could crossconnect them using the OCS and simply “hand them back” when the Routing team needed them. Loaning ports also worked in the other direction when needed. As a result, there was no re-fibering, and no messing with something that was already working – just a click of the mouse instead.
“Using the Calient S320 for testing forty 100G WAN circuits, with a limited amount of 100G test sets, saved valuable time not having to move, scope and clean fiber. We were able to test more circuits with less staff and did not experience the normal downside associated with moving fiber. The CALIENT switch was a major factor in our success during SC18 and SCinet looks forward to utilizing the S320 in the years to come.”
Kevin Quire, Deputy Co-Chair, SCinet WAN Team
Making 400G History
CALIENT was also a key part of a history-making event at SC18 in which production traffic from national and international networks was successfully passed to Dallas using a single, 400 gigabit Ethernet link over metro distances of approximately 6 miles. Traffic ran error free through a CALIENT S320 optical switch between two Juniper devices: QFX10003-80C and PTX10003-80C. See Making History at SC18: Production Traffic Passes Through First 400 Gigabit Ethernet Metro Link in Texas
All in all, SC18 was a great event for the industry and for CALIENT, and I’m really looking forward to helping to plan next year’s event which will take place in Denver. Now that the SCinet team is accustomed to some of the benefits of using optical switching to automate and simplify testing, I’m hoping that we can demonstrate a whole new layer of value as part of the live production network connecting within the data center environment and to the WAN-edge.
Editors Note: Aaron Hudson is a Senior Engineer in CALIENT’s Systems Engineering Team.