Some days I can't decide if I love managed firewalls providers and their employees that my client doesn't have to hire, train & manage, or if I long for the days gone by when the client's local W2 IT employee could be "blamed" for firewall problem.
Fortunately modern software technology can help IT Project Managers such as myself (who usually can't even spell "IP") find problems in a firewall that either the local IT guy or the managed firewall contractor are fussing over (and pointing fingers at each other).
"What Was The Last Time it Worked?"
This is the first question to ask anyone reporting a problem in any ICT (information & communications technology platform).
The second question is, "Who changed what since it last worked correctly?" Unfortunately, the usual answer to question two, is "Nothing" or "I don't know".
Once you have two instances, one where the application worked and one where it didn't, all you have to do is "Spot the Difference" in the underlying application programming to determine "what changed" so you can focus in on repairing the change to eliminate a new problem that's being reported.
MSP Secret: Archive Copies of Your Firewall Configuration File
Your business firewall is an active and evolving security force ensuring that the "enemies" on the outside and your "frenemies" (employees) on the inside of your business communications network don't mess you up.
In order to achieve this protection the firewall is being constantly updated which makes changes to the "configuration file" of the firewall. This "config file" contains all the rules that allow one sort of access while blocking others. Whenever an employee reports "the internet's broke" they are likely reporting a problem in the managed firewall or a problem in the VLAN of a managed switch.
If you use a "managed services provider" or "MSP" like Spectrotel to manage your firewalls & switches at least you know "who's fault it is" if the source of the problem is "inside" the firewall versus "outside" the firewall. ("Outside" firewall problems usually involve the cables into the various ports being re-arranged).
Unfortunately most business clients don't keep close tabs on what's going on inside their own firewalls. This is a mistake as it's a required "managed firewall best practice" to create archive copies of your firewall's config file so you can easily spot changes before or after a firewall problem is reported.
Use Notepad++ & the "Compare" Plug-in to Easily Spot Changes
I'm currently trying to help my client's IT engineer confirm that a new updated firewall has the correct copy of the old fire wall's config file (with appropriate upgrades) so we don't find out at midnight when we go to swap out the old firewall for the new one.
Shawn Justice, Spectrotel's VP of Ops (and frequent target of both my professional joy and bewilderment) recently share one of his MSP secrets that he uses to "stare and compare" a problem to resolution where he uses Notepad++ and its "Compare" plugin to find differences between two difficult-to-read config files. Shawn emailed me this link to a blog post that explains to project managers like me how to do it.
I read the post, downloaded Notepad++, activated the "compare" plugin and was comparing config files in under 5 minutes.
"So What Am I Supposed To Be Doing Again That Helps Me Manage My MSP?"
1. Request "read only" access to your managed firewall
2. Download a copy of the firewall's config file on a periodic basis after getting a "Day 1" copy after turn-up
3. Use the Notepad++ "compare" plugin & software to easily find changes in your firewall config files from one week to the next.
4. Known changes are where your start asking questions when your firewall seems to be suddenly having problems
5. Unknown changes can help you spot unwanted intruders.
Carrier MSPs Versus Non-Carrier MSPs
It's worth noting that "carrier" MSPs like Spectrotel approach documentation quite differently than a "non-carrier" MSP like whomever you might be using that charges a monthly fee for "doing all your IT stuff".
Non-carrier MSPs are the "original" MSPs and provide a useful service for any business that need's maximum help both managing and documenting all IT appliances.
Carrier MSPs generally charge no monthly MSP-type fees because their philosophy is that your monthly "rental" of your managed firewall and managed switch pay for the initial setting up of the gear and periodic programming updates. If and when you need a carrier MSP to "get in the weeds" with you to solve an IT problem outside their gear or outside the "handshake" between their gear and your gear, carrier MSPs will bill you a prearranged "professional services" or proserv fee which is usually hourly anywhere between $100 and $500 per hour.
Keep Your Own Local Documentation
One of the reasons carrier MSPs charge less for what appears to be about the same service as more classic MSPs is that carrier MSPs are not going to keep up-to-date documentation of anything and everything. In many cases, the current config file for a fireall is the only config file. If you want your carrier MSP to keep back copies of anything you need to 1) confirm on your agreement that requirement, and 2) frequently check to ensure it's being done.
Write Better Trouble Tickets From Your Own Documentation
Multi-location businesses are usually the best fit for carrier MSPs as multi-location businesses generally have a small in-house IT staff that likes to focus on the headquarters IT gear and then allows the carrier MSP to primarily look after the WAN (wide area network - the internet connections that connect each location) and the VPN (virtual private network - the secure connection between business locations made over the public internet).
An important "best practice" that successful multi-location IT departments adhear to are having their local site managers keep their own "Scope of Work" or "SOW" documents. An SOW is an electronic document that starts out as a blank piece of paper but then contains the "cookbook recipe" of how their managed firewall & managed switch connect to computers on the inside LAN (localk area network) and the outside WAN. The SOW also holds diary entries of all MACs (moves, adds & changes) as well as archive copies of all config and similar files.
When a multi-location local manager needs to report a local "internet problem" much time & money is saved when the local manager can consult his or her own SOW, cite "when it last worked" and then before and after copies of the config file. The local manager doesn't have to understand how firewall's work to do this and the resulting carrier MSP trouble ticket is easy for the carrier MSP to quickly address and resolve.