In my earlier blog post, Troubleshooting OT Security: Why IT-Site1 Can’t Ping OT_Site1R, we discovered the reason for this issue. Our “who done it” is simple. For security reasons, we are using Cisco TrustSec to keep them from communicating. Which is pretty wicked cool since they are in the exact same subnet and broadcast domain.

I mentioned at the end of that blog, that I’d follow up with a blog with sniffer captures and also look at the show commands that deny this traffic in the CLI.

Ready to play? Let’s go!

Show and Tell Time: PCAPs and Show Commands

The Ping and the Packet Flow

First let’s think about the flow of the first PING (ICMP echo request) through the eyes of Cisco’s TrustSec of classification, propagation, enforcement

1. Classification: The ICMP Echo packet from IT-Site1 comes into Site_1-3400-2 and gets classified. In my earlier blog post, Troubleshooting OT Security: Why IT-Site1 Can’t Ping OT_Site1R, we showed that it is classified. It gets classified in Group IT. The SGT assigned to it is 401.
2. Propagation/Transportation: Hop by hop, on the way to the destination, the packet and it’s SGT, will get propagated/transported
3. Enforcement: Site_1R-IE3400-1 will do the enforcement. It will
   • Look at the packet’s SGT
   • Look at the destination IP address that packet is trying to get to and what security group it is in (SGT)
   • Next it will look at whether traffic from the source’s security group (SGT) is permitted to “talk with” the destination’s security group (SGT)

Time to check out what is happening on the wire: PCAP style!

Time to Have Fun!

In my earlier blog post, Troubleshooting OT Security: Why IT-Site1 Can’t Ping OT_Site1R, we already covered classification. It gets classified in Group IT. The SGT assigned to it is 401. Let’s see what happens on the wire and also look at show commands.

Sniffer Traces:

We will take 2 sniffer traces on Site_1-9320-1.

1. Sniffer Trace #1: Inbound on Gig1/0/2
2. Sniffer Trace #2: Outbound on Gig1/0/12

Sniffer Trace #1: Inbound on Gig1/0/2 of Site_1-9320-1

What we expect to see in this sniffer trace is that the Ping (ICMP echo) from 10.10.11.9 destined to 10.10.11.8 was classified on Site_1-3400-2 and security group tag (SGT) 401 got pushed on. Below we can see that this is what happened.

Sniffer Trace #2: Outbound on Gig1/0/12 Site_1-9320-1

You might be thinking at this point, “Um, Fish, why did you take a sniffer trace going out of the interface? It is just going to show the SGT of 401 also”

Well, yes. I can assure you that we are going to see that. But in my next blog it won’t have it on there going out towards the Cisco URWB. 🙂

So what have we seen?

1. 401 as the SGT on the wire for source IP 10.10.11.9 as it comes in to Site_1-9320-1
2. 401 as the SGT on the wire for source IP 10.10.11.9 as it goes out of Site_1-9320-1 and over gig1/0/12 (vlan 11) towards Site1R over the Cisco URWB.

Can we safely assume that, for sure, the SGT of 401 is still there as it ingresses the IE3400 in Site1R? Well we could do another sniffer trace. But let’s do something else instead.

The Design

IF the SGT made it to the IE3400 in Site1R. That is, if it was properly propagated/transported all the way through. Then, it should be dropped at the remote IE3400 which is doing the enforcement. Right? Why?

Because of our Cisco TrustSec implementation. The design aims to prevent communication between IT and OT using Cisco’s TrustSec. This approach ensures they do not communicate with each other. We can see from the matrix in ISE below that source IT_Mgmt with SGT 401 is restricted. This source is not allowed to talk with devices in OT_Admin with SGT 150. And, spoiler alert, 10.10.11.8 is getting SGT 150.

Let’s go to the remote IE3400 and see what SGT it has classified for IP address 10.10.11.8.

As we can see above the SGT for 10.10.11.8 is 150. In the matrix above we see that traffic sourced from SGT 401 should not be allowed to talk with destinations in SGT 150.

The Enforcement

Let’s go to the IE3400 over in Site1R.

IF the SGT of 401 is still on the packet, the remote IE3400 should deny this. It should also increment counters. This happens for each packet sent from SGT 401 to SGT 150 and denied by the enforcement.
Let’s play!

1. Go to the remote IE3400 and clear the cts counters. Issue the command show cts role-based counters after that.
2. Go to the IT-Site1 and kick off the ping. This will fail as we already saw in the earlier blog.
3. Next, run the command show cts role-based counters again. After running the command, we can see that we denied the 4 ping attempts from 10.11.11.9 (sgt:401) to 10.10.11.8 (sgt:150)
   

Ready to see what number 3 looks like on the IE3400 at Site1R after the clear and the ping? 🙂

Woot! Woot!
And there you have it! Cisco’s TrustSec is securing our environment. It keeps our 2 devices from communicating with each other, even though they are in the exact subnet!

Categories: Network Detective, Security, Troubleshooting