QA rule zero: whatever changed last caused the problem.
At 6:00 am, my wife woke me up to inform me that she could not access her Gmail. Ironically, I switched my parents, kids, wife, and office to Gmail in part to minimize tech support calls. And even though it’s a holiday (MLK Day), tech support takes no holiday.
Here is the Verizon FiOS Internet error message (click to enlarge):
And, for those trying to Google this problem (presumably from a non-FiOS network), here is the text of the above error message:
http://192.168.1.1/index.cgi?blocked=http://www.google.com/reader
Attention
Access Blocked: Your attempted access to URL http://www.google.com/reader was blocked (Keyword Filter). Contact your network administrator for help.
As error messages go, this one is pretty good. We have the URL starting in 192.168.1.1, which is the IP address of the Verizon FiOS router (an Actiontec MI424WR in my case). And we know that a “Keyword Filter” is causing Google to be blocked.
In an earlier life, I did quality assurance (QA) on both hardware and software (at BBN, if you’re curious). Whenever the hardware engineers changed the hardware and broke the system, they suspected bad software. Whenever the the software engineers changed the software and broke the system, they suspected bad hardware. I learned if you broke a system that was working before you touched it, then whatever you changed last is what caused the system to break. Software engineers broke software, hardware engineers broke hardware.
Other good practice that I learned from working in QA:
- Take good notes.
- Figure out what works.
- Figure out what doesn’t work.
- Make an educated guess about what is causing the difference between what works and what does not.
- Test each possible cause one at a time. Do not test more than one variable at at time.
- Lather, rinse, repeat.
- Take good notes.
I tested other TCP/IP applications. I was able to VNC over SSH into my work computer. Therefore, the Internet connection was working.
I was able to access Gmail via a secure connection (e.g. https://www.gmail.com/) and was able to access other websites via HTTPS but not via HTTP. The HTTP protocol runs on port 80, the HTTPS runs on port 443. (See the IANA well known ports and registered ports list for details.) I have heard of ISPs blocking inbound access to port 80 to discourage home users from running web servers, but I have never heard of blocking outbound HTTP traffic.
Here I cheat a little. I recognized the error message as being from the Actiontec router’s built-in “parental control” software from the 30-day free trial I received in the fall of 2006. But I’m getting ahead of myself.
To login to the router, point your browser to http://192.168.1.1 and enter the default username (admin) and the default password (password1). If your Verizon FiOS installer gave you different login credentials, then use those. I had to ask the installer for this info. And although you can customize the router, Verizon will reset it to factor defaults if they have to make a service call.
I navigated to the Parental Control page and tried to create a new rule. Sometimes broken software behaves like broken hardware: all you need to do is turn it on, turn it off, and then it works. But in this case, I could not delete the test rule that I created. Nor could I edit it. The router kept hanging whenever I tried to do so. I did notice that the SurfControl icon that had appeared on the Parental Control page was no longer there. In the fall of 2006 (when I first got FiOS), you got a free trial of SurfControl and then had to pay to keep SurfControl active. Perhaps Verizon struck a deal with SurfControl to provide filtering software for free. Perhaps the deal with SurfControl has been terminated.
QA rule zero: whatever you changed last is what caused the system to break.
I had changed nothing on any computer from the time Gmail last worked until Verizon inexplicably started blocking it (and the rest of my web traffic). So I assume that Verizon changed something that broke my router. Perhaps they were pushing a software update out their FiOS customers. Perhaps a software update related to the SurfControl parental filtering software. When I worked at Verio, we often updated the network over holiday weekends. Then again, we also told our customers what we were doing. And if you called Verio, they had a clue about any problems you discovered. Not so with Verizon.
Keep in mind what I know:
- HTTP (port 80) traffic is blocked.
- HTTPS (port 443) traffic is not blocked.
- Other TCP/IP traffic is not blocked.
- Parental filtering software is acting flakey.
- I have changed no hardware and no software since everything was last working.
Here’s how my call to Verizon “tech support” went:
ME: My wife logged on this morning and was unable to access Gmail.
VERIZON: What browser did you use?
ME: Firefox. (Note to self: always answer “Internet Explorer” when asked this question.)
VERIZON: Try it with IE now.
ME: OK [said he, trying not to act like he knows infinitely more than the tech support drone]. Nope, still doesn’t work.
VERIZON: Can you access the Internet at all?
ME: I can access Gmail securely via HTTPS but not via HTTP.
VERIZON: That’s a Gmail problem. You’re going to have to contact them for help.
ME: No, no, no. This happens with any website. Even with www.verizon.com. The URL of the error message starts with 192.168.1.1.
VERIZON: That’s not our network.
ME: What? Of course this is your network. This is a VERIZON error message from the Actiontec FiOS router that VERIZON installed. Something is wrong with your router.
VERIZON: That’s not our network. You’re going to have to bypass the router to test the Internet connection.
ME: I already told you that the Internet connection is working. Only certain protocols are being blocked.
VERIZON: We’re not blocking anything.
ME: You are not listening. I’m going to take the phone and bang it on the desk now. [Takes phone off of ear, bangs it on desk, resumes talking.] The Internet is working. I have a separate window open on a secure connection to my office network.
VERIZON: Do you mean you have a VPN?
ME: Well, sort of. It’s a software VPN, VNC over SSH.
VERIZON: That’s an unsupported network configuration. We don’t support VPNs!
ME: What? No! That’s got nothing to do with it. The filtering software on the router is broken. Do you even understand how the Internet works?
And then the Verizon tech support drone hung up on me. Maybe I was being overly optimistic in thinking that I could get an intelligent answer out of Verizon. Maybe I shouldn’t have mentioned the magic bad words “VPN” or “blocking” which caused the tech support drone to read another scripted response. One thing was clear. Verizon was completely clueless.
I suspect that Verizon pushed a software update out to my router last night, that the software update had to do with the parental control software provided by SurfControl, that the software update had a bug, and that the result was the parental control software blocked all outbound web traffic. I doubt I’m wrong.
So here’s the solution. Login to your router. If you have customized any settings, then print out those pages (or save them as PDFs). Go to the Advanced page and select “Restore Defaults.” The router will give you a chance to save your config file. Save it. It’s a plain text file and it’s full of good info. Then restore any custom configurations you’ve made. And this will fix the problem.
Happy holidays.
Related Posts
- My First Podcast
I’m so proud. - Verizon FiOS TV Problems
How to spoil a movie night in one easy step. - Happy Happy FiOS FiOS
Over the last 17 years, my Internet connection has gotten 1500 times faster. - Getting FiOS
Fiber comes to the home.
