I’ve been having an absolute blast playing in a lab environment for OT Security. It is this wicked cool setup I got the great fortune to play, I mean work, in. So much more is coming and I admit I am finding myself incredibly intrigued by the OT world.

The Case: IT-Site1 Can’t Ping OT_Site1R

(Despite Being in Same Subnet and VLAN)

The “Routing”

Well the “routing” seems pretty much a done deal since they are both

• In the same VLAN – 11
• In the same Subnet – 10.10.11.0/24

Leaves very few suspects here. Something is wrong with Layer 1 or Layer 2. Or maybe the masks are incorrect on the end devices trying to ping each other.

• Mask is incorrect on the end hosts
  • Nope. Masks are /24 on both
• VLAN 11 isn’t going across the Cisco URWB (Cisco Ultra Reliable Wireless Backhaul)
  • Site_1-3400-2 (10.10.11.88) can successfully ping directly connected IT-Site1 (10.10.11.9)
  • Site_1-3400-2 (10.10.11.88) can successfully ping OT_Site1R (10.10.11.8)

“Routing” has been cleared as a suspect. By doing the above ping across the Cisco URWB, that too has been cleared as a suspect.

In these Troubleshooting Ride Alongs I will often be giving you a fixed “suspect”. This is one of those types of blogs. The “who done it” then was in the below suspect list.

• Routing
• Cisco URWB
• SGTs

Routing and Cisco URWB have been “cleared” of having any part in why the two end hosts can’t ping. So that leaves only ONE suspect: SGTs

The SGTs

If you have absolutely NO idea what an SGT is, I recommend watching my friend Katherine McNamara’s YouTube video. It is titled “TrustSec: Overview of TrustSec and Terminology“. She does a nice setup and then at 5 minutes and 30 seconds in she goes to a whiteboard. 🙂

……Now back to our setup.

Look at the diagram above. To the right of IT-Site1’s IP address 10.10.11.9 there is a (401). To the right of OT_Site1R’s IP address 10.10.11.8 there is a (150). These are the values/tags (SGTs) that represent the security groups I want each to be a part of.

• OT-Site1R should be in a security group. This group should include a subset of other OT device. The tag (SGT) for this group has been defined to be 150
• IT-Site1 should be in a security group. This group should include a subset of other IT devices. The tag (SGT) for this group has been defined as 401.

How do you get a device into its respective “group”? That would be a blog all onto itself.

For this environment

• A static mapping of 10.10.11.0/24 to SGT 150 is configured in the 2 IE3400s the hosts are cabled to
• The config for this is cts role-based sgt-map 10.10.11.0/24 sgt 150

At this point you should be saying something like “wait Fish..huh? … but IT-Site1 is in 10.10.11.0/24 and you have in the diagram that it is supposed to get 401.

Easy – that device is getting assigned the SGT of 401 dynamically based on its identity.
Let’s play.

• Shut the interface gig1/7 on Site_1-3400-2. This is the interface that IT-Site1 is on.
• Turn some debugs in Site_1-3400-2
• No shut interface gig1/7
• Watch the magic

We do not understand exactly what is going on above. Still, we can try to reverse engineer a little.

• Gig1/7 comes back up
• Something about dot1x and mac=0050.5698.30f3. (side note: this is the mac address for IT-Site1)
• Some stuff in relationship to CTS (Cisco TrustSec)
• Things start seeming to happen with some CTS-SXP (Cisco TrustSec Security eXchange Protocol)
  • 10.10.11.9/32 add true SGT 401

So clearly Site_1-3400-2 is communicating with something outside of itself and getting informed that 10.10.11.9/32 with mac=0050.5698.30f3 has been earlier identified to be in a different Security Group – one with SGT 401.

I mentioned earlier that IT-Site1 was receiving an SGT different from the statically configured 150 for its subnet. I also explained that this should happen based on its actual identity. It should not solely rely on the subnet that it is in.

We are using Cisco’s Identity Services Engine for this identity lookup. As I mentioned, to keep this simple, I just did the identity based on mac address.

Case Closed: Cisco’s TrustSec is Securing Our Environment

Yes, IT-Site1 and OT-Site1R are in the same VLAN. They have been identified as being in two distinct Cisco TrustSec Security Groups. These groups are IT and OT. And for this design we do not want IT and OT being allowed to communicate with each other.

In my next blog, we will do a sniffer capture. We will see the SGTs. We will also look at the show commands that deny this traffic in the CLI.

Hope you had FABULOUS FUN in this Troubleshooting Ride Along! I know I had an absolute BLAST building this for you!!!

Categories: Network Detective, Troubleshooting