Since CES 2010, Verizon has been hard at work launching 4G LTE products and expanding their 4G coverage, but it's hit a few bumps in the road as of late. The month of December was particularly bad for Verizon and its customers who have taken on 4G devices expecting the same rock solid coverage they've always had with Verizon 3G service. Verizon Wireless's VP of network engineering, Mike Haberman, took the time to break things down for GigaOm (and the rest of us via a press release) and explain a little of what has been happening behind the scenes at Verizon to cause the recent issues.
Starting with the December 7th outage caused by a failure of its back-up communications database, Haberman then noted December 21st downtime that was caused by a portion of their IP Multimedia Subsystem (IMS) — which replaces the old signaling architectures used in 2G and 3G networks. Finally, the most recent failure on December 28th was caused by was caused by two IMS elements not communicating properly.
In all cases, the IP Multimedia Subsystem was the root cause but once fixed the same issues never occurred again. In other words, they were presumably isolated bugs in the system which are bound to be a part of expanding a nationwide network of this nature. "Being the pioneers, we're going to experience some growing pains," Haberman said.
Haberman isn't hiding from the issues and notes "These issues we've been experiencing are certainly regrettable but they were unforeseeable." and highlighted Verizon's goal which "is to ensure that our 4G networks meets the same high standard that our 3G network does, we’re not there yet, but we’ll get there."
So does Verizon deserve some slack here? That's a matter of personal opinion. Reality is, launching a new nationwide network with bleeding edge technology is hard and there's bound to be some bugs that are destined to wreak havoc. In these cases, the errors found within the system caused issues for many users but in Verizon's defense, you can't really place a "beta" tag on a network so to speak. However, you can stop introducing ridiculous fees and messing with upgrade policies that upset customers when some are already upset with their services.
Reader comments
Verizon: LTE is hard, y'all
I dont recall sprint having these Network issues with their 4g network and they are the pioneers of the first 4g network in the US. Typicall verizon
As I have heard the problems have all been in the core switches, which are provided by a vendor. They are now swapping the whole solution out with dedicated point to point pipes as opposed the set up now that is failing during back ups and fail overs.
it is just one of those things that happens in IT with complex solutions. Something [i]should[/i] work, and just doesn't.
I'm willing to give Verizon some credit and some slack on this one. I like the fact that they have described what the problem was and that their fixes for the most part have worked. Now if it were the same problem over and over again I'd be worried.
I agree that the announcement of the $2 bill-paying charge was really bad timing considering the recent 4GLTE connectivity issues they're having. But let's face it, 4GLTE is a new technology. There were bound to be some bugs to exterminate. We are all part of the grand experiment, like it or not. They could only simulate and de-bug the service for so long. Real-world testing with millions of real users on their 4GLTE network was inevitable. I don't like the outages, either, but Big Red does appear to be trying to address the issues, and at least they are being honest about it. But like it or not, we will ALL be assimilated into their 4GLTE network in the not-so-distant future.
Resistance is futile...
I also found their "Community" website is down this morning. I wonder if they are getting slammed over the fee and the 4G outage?
Actually the amazing thing if you read the full Gigaom article is that it seems Verizon attempted to run their LTE network nationwide without regional backups...that is amazingly stupid. So instead of a problem cropping up in Idaho and staying there the entire network gets whacked.
What ever engineers that came up with that bright idea should be looking for new jobs.
Something is fishy here...
WiMax is a similar OFDM wireless tech and Sprint never had any major issues with it.
Verizon customers should be very careful moving forward. Verizon rushed it out way too soon and it's starting to show.
The way it is implemented it different if I remember correctly from my time playing with an Epic, more like WiFi.
It's not the wireless failing here, it's the tech enabling CDMA and LTE to work with each other.
EDIT: Looked it up. IMS runs up on top of eHRPD enabling more functionality than just basic handoffs. It's actually required to do VoLTE and SMS on LTE.
I have sprint, and in Sacramento. Wimax has never had an outage. I know it's not lte, but if it went down I'd give em some slack cuz they give me unlimited everything. 3g has gone down twice and each time they credit $25. That made it worth having to call for me.
WiMax was more mature by the time Sprint/Clear rolled it out. Not a great comparison.
There are so many different departments in Verizon. The LTE/Network department and the billing department probably aren't even based in the same state let alone actually talk. It comes down to a PR issue, and as you can see they don't talk to any other department.
Bottom line is every customer should write a check and mail it to them now. Let them take the time and manpower to open the envelopes.
Or go to a corporate store and pay in change and watch the sales rep count put 100 dollars in pennies.
Those things will cost them much more in lost time than the 2 dollar fee they would make money on.
I'm so on board with you. That would be epic.
I'm guessing that Verizon doesn't process the checks themselves. You send the checks to a postal drop box, and a service picks them up and processes them. I worked a job like this in college as a temp, and the local dropbox was for customers of major retailers all over the area, plus the big electrical utility.
Even if you use a pay kiosk in the store, an armored car picks that up (because it's a mixture of cash, and checks) and takes it to a bank service center, more than likely.
There is no way that Verizon should have pushed out so many devices on LTE if the network was so fragile. The rollout should have been relaxed and, yes, perhaps marked beta - or, at the very least, not so aggressively marketed.
Cut them some slack? I don't care about the $2 fee - I've been using bill pay from my bank for years, so I don't really care about that. I don't even care all that much about data caps - I do think that there could be more tiers, and I am glad to have grandfathered unlimited, but the truth is that I rarely use more than 500 MB a month anyway. However, when Verizon spends so much time and energy selling devices that depends on a network that they seem to have no idea how to keep up in a stable way - sorry, Verizon has earned no slack. They have earned a front page notice every time the network goes down.
Hmmm, maybe AT&T will come out with a map TV ad of their own, with a huge asterisk over Verizon's map? Or have it blinking on and off...
So this is why they need another $2 a month? To hype a network that has been down 3x this year nationally. Not just a simple local outage, a national one. They should be refunding some of the extra fees already charged to the new 4G plan holders.
actually it has been down 3 times this month and if I am not mistaken about 5 or 6 times this year (once in April, May and I believe one time in October).
Yes the prorated amount. If they want to charge me to pay my bill then why should they not have to refund me for the services that I paid for that they DID NOT provide?
It's the principle here!
Everyone complaining about 3 days of no 4g lte coverage...I get y'all ...specially when your like me....just opened up service with them...and expierence 2 outages within a week...but really..u want them to give u aboercentage back? Come on.... $30 data plan 4gb.....divided by 30 days....pro rated refund would be around 3 dollars....that's what y'all want? So don't be expecting 30 full dollars or anywhere close to it
You're not looking at this from Verizon's perspective. A $3 refund to you is nothing.. a homeless man on the street could spare that.
Take that $3 and multiply it by their 90+ *million* customers, and you can bet they'd have real motivation to ensure this kind of thing simply doesn't occur. Verizon would start isolating the network by country region, and invest in failover redundancies. Stuff that companies with less critical networks do.
I think Verizon will eventually get a handle on it. Right now, there's no fire lit under their collective ass because their sterling reputation and top-tier status to go with it protects their position. Customers can bitch and whine until they run out of oxygen, but if they really believed they could do better, they would. Verizon already understands this.
So long as these issues don't become a habit, nothing will change, and so they will continue to strip customer advantages and apply more restrictions on phones and contracts. Verizon knows people will put up with it because "everyone knows" they have the best network. If nothing else, the notion that they can continue get away with that is motivation enough for them to beef up their network.
I was just told by a verizon customer rep that they are still having issues with the 4g service here in NY. I'm sure this isn't the only place either. They are pushing these amazing phones yet they can't even properly support them.
I think they deserve some slack. Not a tom of slack, but yes, some. They're not hiding from the fact that there are problems with the 4G network which is a plus in my book. But they are the pioneers of a new technology so I'd expect some minor issues. Its not like the outtages have lasted weeks on end. But the reason I am cutting them some slack is because 4G is new, but they don't charge a premium over 3G. And at least for me, when the 4G went out I still had 3G coverage. So I'm ok with being a "beta tester" for 4G since I don't pay extra for it and don't completely lose data coverage if its not working.
It's pretty hard to hide an issue that affects your ENTIRE LTE customer base in the country. :)
Not really. If they were Apple they would find a way to blame it on the user or try to convince users that it really is working and they are looking at it wrong.
LMBO...that is funny and so true. That is why I am using a Thunderbolt and not an iPhone!
Not really. If they were Apple they would find a way to blame it on the user or try to convince users that it really is working and they are looking at it wrong.
Maybe Verizon will start refunding customers when they have outages, being as they are charging for tech support and online payments. Fair is fair, right?
Yeah there should be a law that any service provider needs to refund customers for time that they are unable to provide that service. If the network is out 3 days out of 30 then I should get that percentage back.
If they are forced to do that every time the network went down, to every customer, you can bet the outages will be resolved.
I'll bet that anyone that was truly impacted by the outage could get some consideration on their bill if they called.
I think his point was that you shouldn't have to call in the first place.
You should call i have been a sprint user for years, any time i have had service issues (pretty much all the time) Id call & tell them i had call drops poor or no data they send me a little kick back. Not much but the gesture is nice. I always felt they were keeping up the their end of the deal. Now i switched to VZW i hope they at lest lube me up and hand me a towel at the end of the night.
exactly....you shouldn't have to call. Same thing with comcast. If my cable service is out, they will give me a credit for the days of an outage, but I have to call. Annoying. The record of me calling to say the cable is out should be good enough. But corporate greed knows that most WON'T call for a credit, so they get to keep it all.
My daughter called and they gave her $1.97 per day it was out. I have heard that some people got $15.00 back (credited to their account).
Wait a minute, verizon charges for tech support? How much?
Agreed. Maybe they need to put more effort and focus on keeping the network stable and supporting the handsets they release instead of trying to find new ways to f++ k us out of money and release a new handset every other week.
Their support for handsets once they are released is pathetic and they should be embarrassed. Perfect example of that is the TBolt. Nothing but software issues, horrible battery life, connectivity issues and empty promises since release. The only reason the phone was usable was a result of root and custom ROMs.
The premium that we pay as vzw customers is starting to not be acceptable anymore as a result of the BS fees and constant network outages.
Should me paying my bill be something that any company charges a fee for? No way. That's THEIR cost of doing business not mine
Let's see how they like it when I go into the store to pay my bill in pennies every month.
At least if you paid with pennies they would get all their money. When you pay with a card they pay a fee to the card issuer. That is the reason for the $2 charge.
Credit cards have always been like that.
Yes they have, and it makes no sense that if you use that card to set up regular automatic payments, there is no fee, but just calling and making a one time fee with the same card gets you hit with the fee. Makes no sense. They've reversed course so it's a moot point, but it's corporate greed at work. Don't spout that capitalism crap. WHY do things need to grow every year? If VZW made a $1billion dollar profit in a year, why do they need to grow that the next year? $1 billion won't be enough next year? Maybe if corporate execs brought their salaries back in line with what they were in the 60's, the Middle class would be relevant again, instead of shifting the wealth to the top percentage.
They would deserve slack if they didn't charge $2 for online payments and didn't cap people's data at 2 GB/month.
They are charging 2 dollars because the government capped the amount that banks can charge for wire transactions... banks in turn pass the cost on to Verizon... Verizon in turn passed the cost to you... its economics.
Same reason why banks started charging debit card fees..
This is why you vote for ron Paul
Ron Paul is a idiot sir... my argument is based on logic... your comment is based on ignorance..
I so agree. But, if we vote Ron Paul in....we might go to World War and then we won't need Cellphones because we will all be dead!
Most of the banks that decided to charge debit card fees have since changed their minds. Why? Because consumers decided to switch to institutions that didn't charge debit card fees. It's all about competition. The fact of the matter is that the carrier market is not sufficiently competitive.
I disagree with you. I`m with Sprint and I never been charged to pay online by CC or other way... It`s just VZW because they are the best as everyone of you claims....
talk about silly fees, does sprint still charge those fees for 4G devices, even if you live in a 3G only area (which is everywhere)
remind me how VZW is not the best again?
I cannot believe Verizon has the gaul to charge for payments on line, but not by mail or in the store (which takes valuable employee time).
So if I was on Verizon, I would go to store and pay after I went to bank, cashed check for the amount in either rolls of change or dollar bills and go stand in line to pay.
After their stores were tied up like this for a month, I'm sure they would get it - their customers do not like this silly charge.
Yes, it would take my time but be worth it to annoy them and force
sensible change.
I don't like paying online. I always go into the store and pay right there. I get a printed receipt and I am good to go!
Yea ok so have a few bugs. Instead of charging 2 bucks to pay
by credit card you give us half off on data for December on each 4g devise we have. I have two