QA rule zero: whatever changed last caused the problem.
At 6:00 am, my wife woke me up to inform me that she could not access her Gmail. Ironically, I switched my parents, kids, wife, and office to Gmail in part to minimize tech support calls. And even though it’s a holiday (MLK Day), tech support takes no holiday.
Here is the Verizon FiOS Internet error message (click to enlarge):
And, for those trying to Google this problem (presumably from a non-FiOS network), here is the text of the above error message:
http://192.168.1.1/index.cgi?blocked=http://www.google.com/reader
Attention
Access Blocked: Your attempted access to URL http://www.google.com/reader was blocked (Keyword Filter). Contact your network administrator for help.
As error messages go, this one is pretty good. We have the URL starting in 192.168.1.1, which is the IP address of the Verizon FiOS router (an Actiontec MI424WR in my case). And we know that a “Keyword Filter” is causing Google to be blocked.
In an earlier life, I did quality assurance (QA) on both hardware and software (at BBN, if you’re curious). Whenever the hardware engineers changed the hardware and broke the system, they suspected bad software. Whenever the the software engineers changed the software and broke the system, they suspected bad hardware. I learned if you broke a system that was working before you touched it, then whatever you changed last is what caused the system to break. Software engineers broke software, hardware engineers broke hardware.
Other good practice that I learned from working in QA:
- Take good notes.
- Figure out what works.
- Figure out what doesn’t work.
- Make an educated guess about what is causing the difference between what works and what does not.
- Test each possible cause one at a time. Do not test more than one variable at at time.
- Lather, rinse, repeat.
- Take good notes.
I tested other TCP/IP applications. I was able to VNC over SSH into my work computer. Therefore, the Internet connection was working.
I was able to access Gmail via a secure connection (e.g. https://www.gmail.com/) and was able to access other websites via HTTPS but not via HTTP. The HTTP protocol runs on port 80, the HTTPS runs on port 443. (See the IANA well known ports and registered ports list for details.) I have heard of ISPs blocking inbound access to port 80 to discourage home users from running web servers, but I have never heard of blocking outbound HTTP traffic.
Here I cheat a little. I recognized the error message as being from the Actiontec router’s built-in “parental control” software from the 30-day free trial I received in the fall of 2006. But I’m getting ahead of myself.
To login to the router, point your browser to http://192.168.1.1 and enter the default username (admin) and the default password (password1). If your Verizon FiOS installer gave you different login credentials, then use those. I had to ask the installer for this info. And although you can customize the router, Verizon will reset it to factor defaults if they have to make a service call.
I navigated to the Parental Control page and tried to create a new rule. Sometimes broken software behaves like broken hardware: all you need to do is turn it on, turn it off, and then it works. But in this case, I could not delete the test rule that I created. Nor could I edit it. The router kept hanging whenever I tried to do so. I did notice that the SurfControl icon that had appeared on the Parental Control page was no longer there. In the fall of 2006 (when I first got FiOS), you got a free trial of SurfControl and then had to pay to keep SurfControl active. Perhaps Verizon struck a deal with SurfControl to provide filtering software for free. Perhaps the deal with SurfControl has been terminated.
QA rule zero: whatever you changed last is what caused the system to break.
I had changed nothing on any computer from the time Gmail last worked until Verizon inexplicably started blocking it (and the rest of my web traffic). So I assume that Verizon changed something that broke my router. Perhaps they were pushing a software update out their FiOS customers. Perhaps a software update related to the SurfControl parental filtering software. When I worked at Verio, we often updated the network over holiday weekends. Then again, we also told our customers what we were doing. And if you called Verio, they had a clue about any problems you discovered. Not so with Verizon.
Keep in mind what I know:
- HTTP (port 80) traffic is blocked.
- HTTPS (port 443) traffic is not blocked.
- Other TCP/IP traffic is not blocked.
- Parental filtering software is acting flakey.
- I have changed no hardware and no software since everything was last working.
Here’s how my call to Verizon “tech support” went:
ME: My wife logged on this morning and was unable to access Gmail.
VERIZON: What browser did you use?
ME: Firefox. (Note to self: always answer “Internet Explorer” when asked this question.)
VERIZON: Try it with IE now.
ME: OK [said he, trying not to act like he knows infinitely more than the tech support drone]. Nope, still doesn’t work.
VERIZON: Can you access the Internet at all?
ME: I can access Gmail securely via HTTPS but not via HTTP.
VERIZON: That’s a Gmail problem. You’re going to have to contact them for help.
ME: No, no, no. This happens with any website. Even with www.verizon.com. The URL of the error message starts with 192.168.1.1.
VERIZON: That’s not our network.
ME: What? Of course this is your network. This is a VERIZON error message from the Actiontec FiOS router that VERIZON installed. Something is wrong with your router.
VERIZON: That’s not our network. You’re going to have to bypass the router to test the Internet connection.
ME: I already told you that the Internet connection is working. Only certain protocols are being blocked.
VERIZON: We’re not blocking anything.
ME: You are not listening. I’m going to take the phone and bang it on the desk now. [Takes phone off of ear, bangs it on desk, resumes talking.] The Internet is working. I have a separate window open on a secure connection to my office network.
VERIZON: Do you mean you have a VPN?
ME: Well, sort of. It’s a software VPN, VNC over SSH.
VERIZON: That’s an unsupported network configuration. We don’t support VPNs!
ME: What? No! That’s got nothing to do with it. The filtering software on the router is broken. Do you even understand how the Internet works?
And then the Verizon tech support drone hung up on me. Maybe I was being overly optimistic in thinking that I could get an intelligent answer out of Verizon. Maybe I shouldn’t have mentioned the magic bad words “VPN” or “blocking” which caused the tech support drone to read another scripted response. One thing was clear. Verizon was completely clueless.
I suspect that Verizon pushed a software update out to my router last night, that the software update had to do with the parental control software provided by SurfControl, that the software update had a bug, and that the result was the parental control software blocked all outbound web traffic. I doubt I’m wrong.
So here’s the solution. Login to your router. If you have customized any settings, then print out those pages (or save them as PDFs). Go to the Advanced page and select “Restore Defaults.” The router will give you a chance to save your config file. Save it. It’s a plain text file and it’s full of good info. Then restore any custom configurations you’ve made. And this will fix the problem.
Happy holidays.
Related Posts
- My First Podcast
I’m so proud. - Verizon FiOS TV Problems
How to spoil a movie night in one easy step. - Happy Happy FiOS FiOS
Over the last 17 years, my Internet connection has gotten 1500 times faster. - Getting FiOS
Fiber comes to the home.
This is some frightening stuff. I have been considering ordering Verizon FIOS for the house we are moving into next week.
Question: It would be a dealbreaker if they prevent you from connecting to your own box over http and ssh from outside. Do they?
Yes I realize it’s dynamic IP and that’s not a problem thanks to dnydns.org et al. Right now we have Comcast and I can log onto my machine as long as it’s on.
I started my career on a level 1 helpdesk (aka hell), here are a few observations.
Rule #1: NEVER, NEVER, NEVER talk “tech” to a level 1 helpdesk.
Rule #2: Level 1 helpdesks are designed to *handle call volumes* NOT to *resolve problems*.
Rule #3: Level 1 helpdesk pay as well as McDonald’s. The only difference is that the helpdesk people can type a little. The ones that can *actually think* are either quickly promoted or leave for a “real tech job.”
Most tier1 reps are rated on the number of “completed” calls. They are given 10 minutes training and handed a script of things to try. If you mention ANYTHING that is not on the list they will log the call as “completed:unsupported” and hang up.
Here are some simple level 1 helpdesk tips:
– It is not the rep’s fault he is under trained.
– Use 6th grade words.
– Get an incident number.
– Use “common” tech words i.e.:
“web page” not “port 80”
“secure web page” not “port 443”
ping and traceroute are (mostly) ok.
– If asked about software ALWAYS say “XP” and IE”
– NEVER mention “exotic” apps and files like “MP3”, “VNC”, “Notepad” or “command prompt”.
– If they are not heading in the right direction, suggest that you may need a “level 2” rep to resolve the problem. (many places “ding” them for escalations.)
– If the problem is not resolved, request an escalation.
– If the rep hangs up, call back and ask for a manager. Give them the incident number and request the the call be handled by a tier 2 rep that can do more than read a script.
– If they refuse, do a little digging and find the area sales manager’s phone number and give him a call.
– If all the above fails, post everything you have done in a public forum (like this) and take your business to a company that gives a *&^%$.
Thank you very much Erik for the solution. I received the same error message on my computer and didn’t even bother calling customer support but instead went on another computer which was directly connected to the router and had internet connection. I really don’t know anything about routers and computers but your solution was easy to follow and solved my problem! 🙂
Thanks, had the same problem started today. Same router and same issues. Thank god Google indexes your page pretty quick so I found it near the top of the search list.
Thanks dude.
I learned long ago that, when calling the drones, just answer their questions with the “right” answer (even if it is not applicable to your problem). The sooner you do, especially when you have an actually advanced issue, the sooner you will get pushed up the Level 2 support (or 3 if you are already there). The hole time make this dude your buddy. The biggest hump though is Level 1 to Level 2, especially if you are a network guru. I find it funny that, no matter what company I call, Level 1 support DOES NOT LIKE TO ESCALATE. However, if he’s your buddy, he’s probably going to be a bit more willing to work with you. So many people just lose their cool and become frustrated with the person and that’s why so many people get “accidentally” disconnected. Have you ever worked at a job that required no real thought and hated every minute of it? These people are you when you were i that position. Remember that and keep your cool 🙂
Tips for talking to Level 1 support:
#1 (and most important) Ask for their employee ID number and extension. If they give you some bs about not giving that information out, immediately move up the ladder to their supervisor (and further if necessary) until you have an ID of the person. Each time you are “transferred” to a new person repeat this step so you have a chain of events should you need to talk to a real supervisor or make a complaint to the company.
#2, if you get far enough to be escalated, do not get off the phone until you have an internal trouble ticket number to refer to. Don’t buy in on the “its on your account” stuff. Things get “lost”.
#3, If you can, find out the number and/or extension of the department your ticket was generate for so you don’t have to go through level 1 hell again.
#4, get a callback time line and callback immediately after you’ve breached the first interval. If they say 24 – 48 hours, call after 24, not 48.
#5, if no progress is being made, call the corporate number for the company and ask to talk to the department head of “X” department where your stuff was handled. You might have to wait a long time for this person to be available, but it will be worth it.
#6, if all else fails, please be sure to file a BBB complaint. If applicable, file a complaint with a US regulatory service (like the FCC for Verison). If it’s an emailed complaint, CC the company CEO, department head you tried to talk to, all of your friends and family, your Fed, State, and local representatives, EVERYONE. Corporations think they can cut costs and put in drones, hence the state that we are in today. The only way to fix that is for people to stop bitching, demand action, and ACT on those demands in any way they can.
AJ you didn’t get the virus from Verizon, you got it because your pc wasn’t protected in some way. Verizon doesn’t guarantee not getting a virus’ even if you get the protection suite from them, noone can. PC software issues aren’t a Verizon issue or any isp for that matter .
I worked at Verizon, I was a drone for over 2 years so I know well about routers being updated and not being told. Feel free to email me any questions and Erik if you get a Motorola NIM on Ebay you can use any router with FIOS plus a little adjustment of settings in the router to redirect traffic to the set top boxes so u get a guide.
Your router probably got hacked and someone put in rules to block your access with port 80 traffic.
Hi, i came across your post today while looking for some sort of help for this Crap they Call FIOS..i am very displeased with it, we have had it now 6 months and i have talked to the Drones at least 3 times a month with issues..Finally had to have one computer serviced because of a Virus from VERIZON thanks guys! now our Main PC in in the shop for the same reason…I can not get any help, My pc locks up on web sites, the screen turns off and then comes back on and NO help from them at all…I can;t wait until my stinkin Contract is up so i can get rid of the GARBAGE!!
thanks
Master Kuro,
Do you speak English? Because honestly, you have no clue how to create a sentence that one can understand.
Erik,
Nice info. Verizon seems to be learning from their experiences. Unfortunately, just not fast enough. I, like you just have to remember that this FiOS is still new.
-Chris
Erik,
I’d very much like to email you personally regarding this post. Very interesting, some similarities to my own situation…..would like to chat.
-Brian