Tips from a Network Detective

Tips from a Network Detective

Put your detective hat on your head and your Network Detective badge on your lapel.  It’s times for another installment in the Network Detective Series.

Are we going on our first “case” together?  Nope.  Not just yet.  🙂

In this series I’m not going to be able to always call out every time my “techniques” and “methodologies” steer me one way or the other on a case.  But over time you will notice there is a thread in there.  A guiding framework and methodology.

So before we go on our first “case” together… I want to pass on to you what are really my major guiding principles for when I’m on a case.  The “TOP” tips that I think have the biggest return on your time investment as a Network Detective.

helpful_tips_74943355

  1. Be Methodical
  2. Know What is Normal (Knowledge is Key)
  3. Get to the “Crime Scene” as Fast as You Can
  4. Have “Crime Scene Maps”  that Help and don’t Hinder
  5. Let the Clues and Evidence Guide You
  6. Learn and Improve

Tip #1: Be Methodical

Detection is, or ought to be, an exact science and should be treated in the same cold and unemotional manner.

-Sherlock Holmes

This is going to be a hard one.  Not only will there be the “who done it” you are trying to solve.  But all the added stress, pressure, emotions and potential team tensions that will play into all this.  Now, more than ever, you need to be methodical.   You need to already know what is normal in your network. You need to already know where to look for the basic facts and what CLI or GUI you will be using.

My “troubleshooting methodology” is listed below.  A key tip? — Take notes and keep track of what you see.

  • Gather the Facts
  • Collect the Cluesmethodology_186886796
  • Follow the Evidence
  • Interview the Witnesses
  • Question the Suspects

Tip #2: Know What is Normal (Knowledge is Key)

Knowing what is “normal” in your network is absolutely critical!  Why?   If you do not know what is “normal” in your network …. how could you begin to even hope to be able to differentiate a “fact” from a “clue” or “evidence”?

  • Know how your Traffic Flows in Your Network.
  • Know what has Changed in your Network knowledge_128416973
  • Know how your Network is Configured to react/respond to varying Failures in your Network
  • Know the “Modus Operandi” of the Devices in your Network.

Fact: The CPU of Router X is 80%Is this normal?  If you don’t know what is normal on this box, you do not know if this is a clue or evidence. You may end up wasting precious time and energy “questioning” this Router when this is merely a fact and not clue or evidence.

Fact: show proc cpu on an ASR1002-x is 1%.  Question: Do you know then that this isn’t having CPU issues?  No. You can’t know that or assume that.  Why?  For this device you need to look at the CPU of the QFP.  If you do not know this, you are missing knowledge that is key.

 

Tip #3: Get to the “Crime Scene” as Fast as You Can

Just like in the world of detectives, Network Detectives benefit from getting on the “crime scene” as soon as possible. While the facts, clues and evidence are still fresh. time_91012643

If you do not isolate your failure domains…. you may be exposed to experiencing cascading failures. So get on the crime scene as quickly as possible because once the cascading starts you will have multiple suspects to identify and that takes the challenge to a whole other level.

Network monitoring tools can be hugely essential here.  Some crimes you might even be able to avoid because you got called in before the crime when there were activities that your monitoring tools labeled as suspect.

Tip #4: Have “Crime Scene Maps”  that Help and don’t Hinder

Are your network diagrams up to date and accurate? Would they “help” as crime scene maps… or would they “hinder”?  Are your network diagrams easy to understand?

Let’s say we are detectives trying to solve a “who done it” in a 1 story house.  The floor plan (aka “crime scene map”) is as we see it below.  The crime was committed in the Lounge.  All the clues and evidence for motive point to a suspect who was in the Conservatory. No surveillance cameras and no witnesses ever saw the suspect in the hallway.

Question: Based on these facts can we “let loose” on the theory we are holding that he did it?

clue_little

Answer: In this situation absolutely not.  Why?  Because remember – knowledge is key.  And there is extremely key knowledge missing on this crime scene map.

So what about the above “crime scene map” is a “hindrance” and not a “help”?    Ever play the game of clue?

There are 2 hidden passages that are missing from our map.  There is a secret passage between the Kitchen and the Study… as well as a secret passage between the Conservatory and the Lounge.  This essential fact is missing from the map.  Armed with these new facts we can now go search the secret passage between the Conservatory and the Lounge and see if there are any clues or evidence we can find.

clue_hidden

Tip #5: Let the Clues and Evidence Guide You

It is a capital mistake to theorize before one has data.  Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.  – Sherlock Holmes

First opinions are crucial, but if the evidence changes, so must the theory.  – Gil Grissom, CSI LasVegas

evidence_259630202

This is one of the ones that seems to challenge just about everyone. Why?  Because we are human.  We come up with theories that end up guiding our decision and our troubleshooting…. instead of letting the clues and evidence guide us.    You are going to have theories.  That is just a fact. But hold them loosely.  Don’t let your theory guide you… let the clues and evidence alone be your guide. If you don’t you are going to waste time and miss essential clues and evidence.  You know what else this will do?  It will be the filter you use when you are asking someone else to take a look at something.  Trust me, it will.  I see this ALL the time. You are trying to “help” but when you tell me the facts, clues, and evidence they will be from a biased perspective because you already have a theory you are holding onto so tightly it is coloring everything.

Tip #6: Learn and Improve

Insanity is doing the same thing over and over again and expecting different results.

time to improve_140732821

Every network outage is a chance to improve the network.  So after a network outage, regardless of how small, ask yourself:

  • Could this have been Prevented?                   
  • What could we do to be better Prepared if something like this were to happen again?
  • How could we have troubleshot faster?
  • How could we have gotten to “Repaired” faster?

What Now?

One of those 6 tips “resonated” with you…. with your network.  Which one?  What are your thoughts? Your ideas?  You can make a difference.  You can help protect the packets in your network. So go do it!

  1. Be Methodical
  2. Know What is Normal (Knowledge is Key)
  3. Get to the “Crime Scene” as Fast as You Can
  4. Have “Crime Scene Maps”  that Help and don’t Hinder
  5. Let the Clues and Evidence Guide You
  6. Learn and Improve

networkdetective


NOTE:  The above blog originally appeared on Packet Pushers, April 2015. It was modified slightly and re-posted here as part of the Techniques of a Network Detective Series.