Today I am a Hunter

Ok my day was starting out like a normal friday. I had things pretty well lined up for my day and what I needed to get accomplished. I got to the office did a quick check of my e-mail and took a quick look at megatokyo. Well then I get a call from my boss who has not gotten to work yet. He tells me that one of our districts is having extreme network problems and that the entire high school has been totally down for roughly 2 days. He wanted to know if I could take the demo from Fluke over to the school and figure it out. I said "of course" I called the school grabbed my gear and I was gone.

My Job is really funny like that. I can totally have my day planned and it can be instanly derailed by a distress call from a school. Oh and just for the record their problem was not our responsibility. What I am about to describe was a favor....

SO I get to the school and instantly I focus on the core of the network and attack. I start using my wonderfull little tool to probe the network and discover every computer type device in the building -- including all wireless gear. The initial results were what I thought. The network was completly saturated with traffic. Boys and girls thats means that a 100 megabyte network was running at a constant 100 MB over every interface. The fluke blew me away. I thought for sure that a machine was infected with a virus and was the source but the fluke told me otherwise. Or actually it didnt tell me anything really. I was getting pretty mad at it wondering why won't it tell me the source of the traffic -- thats its function! Well after I started to look more deeply into their network I realized it was no virus. Thanks to the Fluke I realized it had to be one of two things A malfunctioning switch/hub or a bad Network Interface Card. Now that I knew what I was looking for the hunt was on.

I started disconnecting segments of their net to isolate the problem and when I would find it the source I would continue to drill down deeper and deeper through the layers of their network until finnaly I found the culprit. In a library they had a mini swith. That switch was plugged into itself. What exactly do I mean. Ok picture it a regular patch cable. One end is plugged into port 1 of this little 20 dollar switch. The other end into port 5. This simple SNAFU launched a broadcast storm that brought their entire network to its knees.

Total time to find and eliminate problem -- 2 hours.

So my entire day has been re-written and oh yeah. The hunt is not over. I have another school that its having similar problems of congestion. I wonder what I will find there.