The Minuteman » Internet

Category Archives: Personal

0 to Attacked in No Time Flat

By | | , , ,
6 Comments

So as I’ve mentioned previously I’ve moved to the world of a VPS which for all intents in purposes is much like being self-hosted.  I used to do this stuff a long time ago, I still do it but not nearly as intensively and for the most part my shell-fu has gotten rusty.

I spent the first part of Saturday getting the server setup and figuring out WHM and cPanel, both unbelievably easy.  The biggest issue was making sure I had things locked down.  I just set up this server though, who could possibly be attacking it?

A6WLUZ bandwidth (full)

Bandwidth usage since I turned on the server.

You can see where I turned the server on on the 13th.  Notice that big spike shortly there after, yeah that was a huge influx of traffic.  It caused the server to grind to a halt.  At the time I thought it was related to me bringing up my site since it locked up within minutes and I had tweaked some server settings an thought that caused the instability.

Come Monday morning I have an email from A Girl that she cannot get in and 2 from the data-center that they rebooted the server after it ran out of memory and locked up.

A6WLUZ_load_full

System loading and availability since being turned on.

You cant see it as well except for the latest incident in those images but there is a serious proc-load spike when those bandwidth spikes occur.  I promptly switched from APF to CSF for my firewall so I could gain use of the LFD.  I spent my time installing and configuring it last night.

A6WLUZ Detail

The Proc Spike I had overnight.

 

A6WLUZ

A more detailed image of the bandwidth spikes.

There you can see the proc spike from an an incident last night.  I did a few more tweaks to the CSF and you can see things were better when they tried again about an hour later.  In the middle of all of this I also discover that there is a way to have Apache watch all the wp-login pages for failed logins.  When they happen, block and ban the IP after numerous failed attempts.  This is why I called myHosting lazy and was so pissed about their approach in handling the problem.

If you are a server administrator and want to protect against the WordPress brute force attack it is quite simple, doubly so if you have WHM.

Login to WHM, goto Software-EasyApache.  Follow the onscreen instructions and rebuild Apache but make sure the modsec2 module is selected.  Build Apache.

Once built, log in to your shell and edit /usr/local/apache/conf/modsec2.user.conf and add the following.

#Block WP logins with no referring URL
SecAction phase:1,nolog,pass,initcol:ip=%{REMOTE_ADDR},id:5000210
<Locationmatch “/wp-login.php”>
SecRule REQUEST_METHOD “POST” “deny,status:401,id:5000211,chain,msg:’wp-login request blocked, no referer'”
SecRule &HTTP_REFERER “@eq 0”

#Wordpress Brute Force detection
SecAction phase:1,nolog,pass,initcol:ip=%{REMOTE_ADDR},id:5000212
<Locationmatch “/wp-login.php”>
# Setup brute force detection.
# React if block flag has been set.
SecRule ip:bf_block “@gt 0” “deny,status:401,log,id:5000213,msg:’ip address blocked for 5 minutes, more than 10 login attempts in 3 minutes.'”
# Setup Tracking. On a successful login, a 302 redirect is performed, a 200 indicates login failed.
SecRule RESPONSE_STATUS “^302” “phase:5,t:none,nolog,pass,setvar:ip.bf_counter=0,id:5000214”
SecRule RESPONSE_STATUS “^200” “phase:5,chain,t:none,nolog,pass,setvar:ip.bf_counter=+1,deprecatevar:ip.bf_counter=1/180,id:5000215”
SecRule ip:bf_counter “@gt 10” “t:none,setvar:ip.bf_block=1,expirevar:ip.bf_block=300,setvar:ip.bf_counter=0”

Save the file and restart Apache.  This will help stop the brute force attacks.  If it wasn’t for the off chance of false positives, I’d be good with a perma-ban and dropping that axe like a rock….

Funny story, I dropped that !@#$ing ax on myself tonight.  Most of the other services are watched by LFD and when you get multiple login failures, it drops the ax and hard.  I screwed up logging in and paid the price.  I was just going along minding my own business and tried to login a couple times with the wrong password and bam there I am behind a curtain with some asshole molesting my balls.  Man, when I describe it like that it sounds like my intrusion detection system works for the TSA.

In the mean time the folks I got the VPS from (they’ve been fantastic support wise, unlike that previous host) are looking into trying to figure out what’s causing the load spikes.  The bummer is it randomly happens so it’s a paint to catch in the act. The good news is the past couple slams the server has actually survived so it’s almost there.  Security wise it isn’t a concern, it’s just and issue with service.

Too Little Too Late…

By | | ,
1 Comment

So I got the following email at 1630 tonight.  I know the ball started at 0800 this morning thanks to twitter.  They may have had that many complaints to work through but here is the email I got.

Dear Barron,

My name is CS Rep and I am writing you from Customer Relation Department. Your case was brought to my attention because you gave us a bad review on Twitter. We are very serious about providing you with an exceptional hosting and customer service experience, we would like to confirm that everything is running as you would like. What would it take for us to became the best hosting provider for you?
Your feedback is crucial for our business to move forward. We are still that strong company with quality and products as we continue to invest more into support and service in terms of training and technology.
Do not hesitate to use my direct line (her number) or 24/7 technical support (their number) or simply reply to this email: [email protected].

Thank you for choosing myhosting.com, I hope we can get your positive tweets shorty.

Sincerely,

CS Rep

My world at work is customer service.  So I am always willing to respond so that if they’re actually willing to improve their service they can.  Here is my response.

Hi Olga,

Let me start off by saying at this point I will be leaving myHosting.  I have invested in outside hosting, at best I will retain my myHosting account for exchange email purposes.  That said as someone who works for a company that strives on customer service I’m going to lay out from beginning to end and my perspective on it.

Last month I had regular service issues despite my use of the CloudFlare CDN. I opened ticket #: FNF-528-19240

After multiple back and forth arguments about whether or not my site was hosted with myHosting finally they just blamed CloudFlare, the problem continued but with less frequency.  I just dealt with it.  For the most part the site would immediately come back on a refresh and none of my customers were noticing an issue or reporting one to me.

Then in the midst of this WordPress attack, I started to have issues a little more frequently. I started to get emails from my Customers and I did what I could on my end to fix the issue.  Then it happened, I and all my customers, 4 total, were locked out of their sites.  We were locked out without any email in my inbox of how to fix the issue.  When the solution did arrive after my promptly emailing support it was a solution that none of my 4 customers could implement, much less be feasible for 2 of them.  Despite my efforts in maintaining a secure WordPress site, including plugins to stop the brute force attacks, my site was rendered unusable not just to me, but my customers.  I actually had to disabled Cloudflare thus increasing my exposure to SPAM and other undesirable traffic.

Just so I am perfectly clear.  The actions of myHosting taken to “secure” the websites for which I am responsible resulted in their inability to function for my customers.  It took me at least 12 hours before I could finally get those sites unlocked for my customers.  During this time my ability to handle issues as well as my general perception to my customers was degraded.  Doubly so since while trying to unlock 2 of the site resulted in 500 internal server errors.  Once that error was corrected, cleanURLs was broken due to other changes to the .htaccess files by myHosting.  Instead of just correcting their errors to the files, they dumped them. This made me look like an idiot again when a customer informed me in the morning he was getting 404 errors.

That night I started the migration to a VPS with another company.  I could not trust that myHosting, even in a VPS, would not mess with my files or otherwise cause me issues and heartache.

I will say the shining spot in this entire mess was it appeared that I dealt with one single support representative.  That is ownership and honestly that is what I like to see.  But here is his last email, Don’t blame him though, he was trying to keep the peace and convey your situation. It is a lesson in needing to be seriously empathetic to customers and the effects of your actions as a provider.

Hello Barron,

Thank you for your patience and we are sorry that you are having an unhappy experience with myhosting.com.

Because 90% of our customers are not using Cloudflare for protection or wordpress plugins to stop unwanted access, we implemented this access restrict.
Because you appear to have a very secure webspace, you would most likely to be safe removing the lines that have been added, but this makes your wordpress website vulnerable to this attack, so please proceed with caution and make sure all wordpress user passwords are complex and secure.

We have disabled the .htaccess files on those two websites and they appear to be loading currently. If you would like, remove all the added code and turn your cloudflare back on.

Please let it be known we are trying to protect our customers the best possible way. Because of the urgency of the matter, this was the quickest solution. We hope this does not ruin your experience with myhosting.com.
http://statusblog.myhosting.com/
http://statusblog.myhosting.com/#oncloud

Regards,

Here’s how I read it:

Your Text.
My Corrections in Phrasing.
My mental commentary while reading.

Hello Barron,

Thank you for your patience and we are sorry that you are having an unhappy experience with myhosting.com. Because evidently the idea someone would be unhappy about being locked out of their own website surprises us.

Because 90% of our customers are not using Cloudflare for protection or wordpress plugins to stop unwanted access, we implemented this access restrictdecided to treat all our users like idiotic children that know nothing about anything.  Luckily I have experience with being penalized because of the actions of others.

Because you appear to have a very secure webspaceactually know what the fuck you’re doing and have previously educated our support staff, you would most likely to be safe removing the lines that have been added, but this makes your wordpress website vulnerable to this attacka brute force attack where they just randomly try passwords, so please proceed with caution and make sure all wordpress user passwords are complex and secure.  Why in the name of god do you think I use keypass and generate 20 character password strings, just for the ease in memorization?

We have disabled the .htaccess files on those two websites and they appear to be loading currently, but we broke clean URLs so they’re still not working right, our bad? If you would like, remove all the added code and turn your cloudflare back on.  You mean I can unfuck my websites if I so choose!?  Here I thought you guys were just out to screw me in front of people I support.  And yes I unfucked every one I could as fast as I could, even before I got your permission!

Please let it be known we are trying to protect our customers the best possible way, by nuking the site from orbit by treating our customers like children and blocking their access to their own sites just the same as the attackers. Because of the urgency of the matter, this was the quickest solution, because we were dumb and too lazy to implement deep packet inspection and notice that the brute force attempts always use the same username, admin. We hope this does not ruinare sorry this has completely ruined your experience with myhosting.com.  We didn’t consider the ramifications of how our actions could possible make our customers look in the eyes of their own clients.  We will think about possibly not treating all our customers like children in the future but don’t count on it.
http://statusblog.myhosting.com/
http://statusblog.myhosting.com/#oncloud

Regards,

The same support guy I’ve been dealing with all day. +1 for that.

And yes, this did go up on my blog, this email will be going up as well.  I want you to understand exactly how badly this has cut into me.  I strive myself on customer service and whenever possible I stop what I’m doing to help when there is an issue.  Even when these people do not pay me a dime for my services.  I get an email at 0400 in the morning and if my phone actually wakes me up I will look and go fix the problem.  I take my wife out to dinner and get a text message that the server is down and I spend the rest of dinner cranking away on my phone to fix the problem.  That is customer service, owning the problem and fixing it.  Most definitely you do NOT create problems for the customer and if you do, you fix them 100% and ensure the site is returned to normal.  You do everything you can for that customer to ensure the problem is fixed immediately and any issues created are taken care of or assisted with to the best of your ability.

You must also remember when you do go and do something like that, it has consequences beyond just your visible customer level.  Your customers have customers.  Some have business and those types of actions cost them money and trust.  In this case myHosting caused a question regarding my integrity with my ability to be a provider for website services.  Integrity once lost can never be regained and I find the actions of myHosting down right deplorable given their impact on me, my business, and my customers.

If you have any other questions feel free to ask.

Sincerely,

Barron Barnett

I’m not holding out for another response back, but I figured I’d give them honest feedback. I will say I was kind of happy to get the kick in the ass to go get a VPS.

No BAG Day for Me…

By | | , ,
3 Comments

Instead I owed my child molesting financially irresponsible uncle $600.  So no gun for me, doubly so since I have to pay property taxes soon too.

I did however make myself feel a little better during the tail end of doing my taxes:

PayingMyChildMolestingUncle

I wanted to avoid Turbo-Tax but it was the easiest for what all was involved this year.  At least I had a little fun with it.  I gave them a 1 because I actually used them.

Saying Good-Bye to My Hosting…

By | | , ,
3 Comments

I suppose I could layout the entire email chain that went down yesterday that actually started Thursday night.

Suffice it to say, for those who aren’t aware there is currently a brute force attack against any and all WordPress websites.  Overall this is not the most difficult thing to spot, most of the logins all use the same user name and overall they’re just not that intelligent.  Evidently my current provider, My Hosting, was getting slammed and quite hard.  In an effort to head the problem off at the pass they edited everyone’s .htaccess files to restrict access to the WordPress login page.  This wouldn’t be a problem except they had a default deny so site owners were locked out.  Most definitely that’s not Shiny.

The last email I got in the exchange as I was trying to fix the issues is here, along with additional comments.

Their Text.
Wrong Words My Corrections in Phrasing.
My mental commentary while reading.

Hello Barron,

Thank you for your patience and we are sorry that you are having an unhappy experience with myhosting.com. Because evidently the idea someone would be unhappy about being locked out of their own website surprises us.

Because 90% of our customers are not using Cloudflare for protection or wordpress plugins to stop unwanted access, we implemented this access restrictdecided to treat all our users like idiotic children that know nothing about anything.  Luckily I have experience with being penalized because of the actions of the few.

Because you appear to have a very secure webspaceactually know what the fuck you’re doing and have previously educated our support staff, you would most likely to be safe removing the lines that have been added, but this makes your wordpress website vulnerable to this attacka brute force attack where they just randomly try passwords, so please proceed with caution and make sure all wordpress user passwords are complex and secure.  Why in the name of god do you think I use keypass and generate 20 character password strings, just for the ease in memorization?

We have disabled the .htaccess files on those two websites and they appear to be loading currently, but we broke clean URLs so they’re still not working right, our bad? If you would like, remove all the added code and turn your cloudflare back on.  You mean I can unfuck my websites if I so choose!?  Here I thought you guys were just out to screw me in front of people I support.  And yes I unfucked every one I could as fast as I could, even before I got your permission!

Please let it be known we are trying to protect our customers the best possible way, by nuking the site from orbit by treating our customers like children and blocking their access to their own sites just the same as the attackers. Because of the urgency of the matter, this was the quickest solution, because we were dumb and too lazy to implement deep packet inspection and notice that the brute force attempts always use the same username, admin. We hope this does not ruinare sorry this has completely ruined your experience with myhosting.com.  We didn’t consider the ramifications of how our actions could possible make our customers look in the eyes of their own clients.  We will think about possibly not treating all our customers like children in the future but don’t count on it.
http://statusblog.myhosting.com/
http://statusblog.myhosting.com/#oncloud

Regards,

The same support guy I’ve been dealing with all day. +1 for that.

That final email just kind of shoved me over the edge with absolutely not wanting to stick around.  Seriously, that’s pretty much how I read when it came in.  I didn’t discover the .htaccess issue with clean URLs being totaled until this morning when Sean emailed me.

Seriously +1 to them on ownership for support.  Other than this recent shit storm they’ve been a decent host but I’m biting the bullet and going to a VPS.  Because of my love for Microsoft Exchange I’ll be getting a separate host for email just for the wife and I but at this point I’m downloading sites one by one and moving them to the VPS when possible.

This site will be the first to move and will hopefully be done by early in the evening.  I still need to finish securing the VPS and doing other setup work.

Say Hello To Jules

By | | , ,
1 Comment

DSC_7750_1_2_tonemapped

I said good-bye to Senior and Junior last week and this is their replacement.  I got a bunch of stuff I wanted, didn’t really lose any features, dropped my payment and cut my interest rate in half.

Why am I naming the new truck Jules, here’s a couple million words.

I will say that 6.7L Diesel is a serious kick in the teeth.  I am a little sad to let my manual transmission go, but there are upshots to an automatic and this one has a full manual mode.  Now to fix the boat for this summer.

I will be getting at least one of these wallets to put my insurance and registration info in.  I will probably get one for myself as well.

Now if you don’t get the tie between the wallet and the name Jules, let me enlighten you.

The running winner was Marci, short for Marcellus Wallace until the wife mentioned Jules.  I also thought about using the code names from Reservoir Dogs.

Deep down I really wanted to do Jayne but another friend of mine’s got that one for her truck.

I think I’m alive…

By | |
Comments Off on I think I’m alive…

I thought my sinuses were just irritated last week from them starting to plow up the fields.  Evidently not.  I went down like a sack of hammers Tuesday, stayed down yesterday and am barely limping along today.

My Ice Cream machine will be off probably for the next bit until I get this sinus issue under control.  I’ve had some bad sinus infections before, but I think this one just set a new record for the worst.  It seriously feels like my teeth are going to pop out of my mouth and that my forehead is currently in a vice.

Go read the folks on the sidebar.  I’ll try and post when I can.  It may not be the usual stuff as I start back up again.

It’s a Weird Feeling

By | | , , ,
2 Comments

Through my online travels I’ve ended up meeting and getting to know a lot of people in digital space.  Most of these people I would have never met otherwise and due to the nature of the online relationship I know more about them than many people I do know in meat-space.

So it’s a weird feeling thinking of someone as a friend that honestly I’ve never actually met in meat-space.  I’ve got plenty of them, including a bunch I have also met in meat-space, but the saddest most helpless feeling is seeing one of my friends in trouble, with not a damn thing I can do.  Especially when it drags on in a manner that doesn’t seem to end.

As another one of my friends said:

She’s going to be just fine. I mean really, whose side do you want to be on? Tam’s or cancer’s?

I betting on Tam.

Jennifer is right, Tam will be fine.  It did however remind me of something that has been discussed before.  The wonder that is the internet and the expansion and alterations to the boundaries of our “tribes”.

There are many who wouldn’t have moved into my circle of friends if it hadn’t been for this invention known as the internet.  It’s nice having it here though because even in the middle of this whole mess I’m reminded of why I love this community and why I’m so glad to be a part of it.

Go give Tam some words of support.  One of these day’s I’ll finally get to meet her, I’ve got some other friends who have met her and have had nothing but nice things to say about her personality in meat-space.

On Robb’s Lack Of Sunshine

By | | ,
3 Comments

So Robb made an interesting observation, one that I don’t entirely agree with.

Me? I see a lot of free guns from the hands of people who starved to death when they realized they can’t eat an AR15.

I don’t see that exactly coming to pass and Scott Adams puts it perfectly.

126195.strip.sunday

That’s the thing, just because you have food doesn’t guarantee your survival   Just because you have guns doesn’t guarantee your survival.  A will to survive and the necessary tools and training to do so on the other hand will be very hard to stop you.