As an Amazon Associate I earn from qualifying purchases from amazon.com

Not a Onerous NOC Life for a Developer at CiscoLive!


Are you a community engineer who’s tinkering with community programmability, however want extra assist implementing your imaginative and prescient for mass provisioning and/or observability?
Or, perhaps you’re a software program developer who’s interested by utilizing your programming, API, database, and visualization expertise in a brand new method with networking, however want product perception to know what’s impactful?

If both of those are true, you is likely to be interested by some background on how we used DevOps fundamentals within the CiscoLive Community Operations Heart (NOC).

Jason Davis NOC CLUS

A pair months have handed since CiscoLive 2022 in Las Vegas, so it appears a superb time to mirror. I had the privilege to take part once more within the Cisco Dwell NOC – specializing in automation, observability, and community programmability. I’ve served multi-purpose roles of speaker and event-staff/NOC for a few years. It’s certainly one of my favourite actions through the 12 months.

Operating a safe, continuous, high-performance community

Preparation for the June Las Vegas occasion began quickly after the New Yr. We start with our base necessities of operating a safe, continuous, high-performance community that showcases product capabilities and advantages. We collect product steering and new-release intent from the assorted product groups and map that towards well-developed community infrastructure designs we’ve used year-over-year that embrace modularity, layered safety, service resiliency and virtualization.

NOC topology at CLUSCiscoLive Community Topology

NOC subject material specialists

The final in-person occasion for the US was CiscoLive 2019 in San Diego. Since then we’ve realized many product modifications. In my opinion, I used to be to train new streaming telemetry capabilities that have been evolving a number of years earlier.
The NOC has a number of subject material specialists throughout the corporate in Buyer Expertise (CX) [includes TAC], Engineering, IT, and Gross sales. I’m coming from the Engineering facet in our Developer Relations group (DevNet). Whereas accustomed to most of our networking applied sciences, I specialize within the programmability, APIs, databases, dashboarding, virtualization and cloud-native applied sciences wanted to connect our business merchandise along with open-source options.

NOC at CLuSTo get the community monitoring elements solidified we depend on a standard DevOps mannequin the place a developer – me – pairs up with subject material specialists in every of the assorted IT domains. I met with area specialists like Jason F on routing, Mike on safety, Chris on wi-fi, Richard on switching and others to make sure correct operational protection.

Acquainted instruments FOR IT Service Administration

We clearly use our business merchandise like Cisco DNA Heart, Cisco Prime Community Registrar, Cisco Identification Companies Engine (ISE), Intersight, the Wi-fi LAN Controller, Meraki Dashboard, Crosswork, ThousandEyes, and the assorted SecureX options. We complement these with different acquainted instruments of an IT Service Administration atmosphere, like VMware vCenter, NetApp ONTAP, Veeam, and specialised instruments like WaitTime. We continued our occasion design by figuring out the configuration administration and provisioning capabilities we wish and think about the efficiency, fault, and telemetry metrics we hope to attain. The performance every device offers is mapped towards these intents and gaps are recognized. At this level we think about workaround choices. Is supplemental info out there instantly by APIs? Can we develop a proximate analog by combining information from a number of sources? Do we now have to drop again to legacy strategies with SNMP and/or CLI scraping?

When we now have a stable understanding of the necessities, merchandise for use, the administration device capabilities and the gaps, we regularly use open-source options to fill within the wants. Because the area specialists are normally centered on their domain-specific administration instruments, the open-source merchandise, programming, and API orchestration actions amongst all of the instruments are the place I focus my energies.

Extra streaming telemetry

As I discussed, I wished to make use of extra streaming telemetry at this 12 months’s Cisco Dwell occasion. We did. Not solely did we use gRPC dial-out with some dial-in, but additionally NETCONF RPC polling for effectivity in some instances. SNMP was not utilized by ANY of our tailor-made collectors – solely with among the business functions.  Because the programmability SME, I meant to make use of YANG fashions with gRPC streaming telemetry or NETCONF RPC, the place out there.  I frolicked reviewing Cisco’s YANG Mannequin Repository and used YANG Suite to check the fashions towards a staging atmosphere.

For instance, the ‘Web Visitors Quantity Dashboard’ confirmed what number of terabytes of visitors we exchanged with the Web.

NOC at CLUSCiscoLive Web Visitors Quantity Dashboard

We liken it to the ‘Tote Board’ that celeb Jerry Lewis used to point out what number of tens of millions of {dollars} have been raised for the Muscular Dystrophy Affiliation charity through the Labor Day telethons from 1966 to 2014.

Creating our dashboard

Our dashboard was created by extracting stats from the sting ASR1009-X routers offering our 100 Gbps hyperlinks to the Web. A customized NETCONF RPC payload, like the next, was requested each 10 seconds.

<filter xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
    <interfaces xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-interfaces-oper">
      <interface>
      <title/>
      <interface-type/>
      <admin-status/>
      <oper-status/>
      <last-change/>
      <velocity/>
      <v4-protocol-stats>
        <in-pkts/>
        <in-octets/>
        <out-pkts/>
        <out-octets/>
      </v4-protocol-stats>
      <v6-protocol-stats>
        <in-pkts/>
        <in-octets/>
        <out-pkts/>
        <out-octets/>
      </v6-protocol-stats>
    </interface
  </interfaces>
</filter>

Instance – NETCONF RPC payload to acquire environment friendly interface stats

As soon as acquired, the data was parsed into an InfluxDB time-series database, then Grafana was used to question InfluxDB and produce the dashboard visualization as above. Utilizing this technique  allowed us to be extra granular concerning the particular stats we wished and diminished the necessity for filtering on the collector.  We look ahead to enhanced capabilities in defining particularly filtered sensor-paths for gRPC dial-in/-out for future deployments.

Persevering with the DevOps theme of collaboration, our WAN SME was eager to get optical transceiver energy ranges on the 100 Gig and 10 Gig Web hyperlinks. We mentioned how If the transmit or obtain energy ranges dropped beneath optimum thresholds, then efficiency would endure, or worse – the hyperlink would have a fault.

Sadly, the instrumentation was not provided on the particular person lane [optical lambda] degree we wished, so we had a spot. Researching work-arounds resulted in a fallback to CLI-scraping of ‘present transceiver’ instructions. Whereas this was suboptimal, I used to be in a position to give you a programmatic technique to ballot the unstructured information and inject it into InfluxDB utilizing Python. The Python schedule library was used to automate this course of each 2 minutes. Grafana dashboard templates have been created to render information queries to our most popular visualization. The outcomes have been helpful as we might see every lane of all optical transceivers and their transmit and obtain energy ranges.

NOC CLUSGrafana dashboard of Optical Transceiver Energy Ranges

Aligning with DevOps

One other DevOps-aligned scenario was managing observability with a number of wi-fi environments. The principle convention community, adjoining resorts and keynote space have been managed by completely different clusters of Cisco Wi-fi LAN Controllers (WLCs). The 4 Seasons resort had Meraki Wi-Fi 6-enabled entry factors managed by way of Meraki cloud. After consulting with the wi-fi SMEs to get their deep insights, expertise, and intent, I reviewed the present state of the administration instruments and portals. To fill in gaps and lengthen present capabilities, I researched the instrumentation and telemetry out there within the WLC and Meraki Dashboard. There was a mix of NETCONF and REST API strategies to think about and we relied on Python scripting to collect information from completely different sources, normalize them and inject into InfluxDB for Grafana to visualise. The outcomes clarified utilization and useful resource distribution.

NOC CLUSConverged WLC-Meraki Wi-fi Consumer Dashboard

NOC CLUSConverged WLC SSO Pair Well being Standing Dashboard

NOC CLUSKnowledgeable Mode WLC Wi-fi Community Controller Daemon (WNCd) Well being Monitor

Placing all of the instruments and assortment scripts collectively offers us a number of observability dashboards; some we present on the video show wall the place you’ll be able to see what we do!

Many different examples of DevOps collaboration and tailor-made observability could possibly be shared, however this weblog would scroll a lot, for much longer!  Should you’re interested by among the Python scripts and Grafana dashboards developed for the occasion, take a look at this Github repo – you would possibly have the ability to repurpose among the effort on your particular wants!

Hopefully this weblog impressed you to collaborate in a DevOps vogue; if you happen to’re a community engineer – hunt down a developer to help make your imaginative and prescient a actuality; if you happen to’re a developer – hunt down a community engineer to get extra insights on how the merchandise work and what instrumentation and metrics are most significant.

For extra info of NETCONF, gRPC, telemetry, examine these studying assets out:

Comply with me on Twitter for the subsequent occasion – I’d be glad to speak to you!


We’d love to listen to what you assume. Ask a query or depart a remark beneath.
And keep linked with Cisco DevNet on social!

LinkedIn | Twitter @CiscoDevNet | Fb | YouTube Channel

 

Share:



We will be happy to hear your thoughts

Leave a reply

Dealssoreal
Logo
Enable registration in settings - general
Compare items
  • Total (0)
Compare
0
Shopping cart