Something is Afoot…

OG-AA794_GRIDAT_NS_20140204171308Back in April of 2013 there was an attack on a power station in Southern California. The attack was calculated, detailed, planned, and execute well. There were many details that perked my interest including the oil tanks being targeted instead of the windings themselves. This would limit catastrophic damage to the transformer. Additionally numerous fiber-optic lines in the area were cut, including those run by Level 3 Communications.

I have read a few writeup discussing the attack and I did come across one theory that was interesting.

Gabriel: Have you ever heard of Harry Houdini? Well he wasn’t like today’s magicians who are only interested in television ratings. He was an artist. He could make an elephant disappear in the middle of a theater filled with people, and do you know how he did that? Misdirection.
Stanley: What the f*** are you talking about?
Gabriel: Misdirection. What the eyes see and the ears hear, the mind believes.
Swordfish movie (2001)

[See the PowerPoint here]

On the morning of the 16th of April 2013 the following events unfolded at, and around, the PG&E Metcalf Transmission Substation in San Jose, Calif.:

  • 12:58 a.m. AT&T fiber-optic telecommunications cables were cut not far from U.S. Highway 101 just outside south San Jose.
  • 1:07 a.m. Some customers of Level 3 Communications, an Internet service provider, lost service. Cables in its vault near the Metcalf substation were also cut.
  • 1:31 a.m. A surveillance camera pointed along a chain-link fence around the substation recorded a streak of light that investigators from the Santa Clara County Sheriff’s office think was a signal from a waved flashlight. It was followed by the muzzle flash of rifles and sparks from bullets hitting the fence.
  • 1:37 a.m. PG&E confirms received an alarm from motion sensors at the substation, possibly from bullets grazing the fence.
  • 1:41 a.m. San Jose Sheriff’s department received a 911 call about gunfire, sent by an engineer at a nearby power plant that still had phone service.
  • 1:45 a.m. The first bank of transformers, riddled with bullet holes and having leaked 52,000 gallons of oil, overheated – at which time PG&E’s control center about 90 miles north received an equipment-failure alarm.
  • 1:50 a.m. Another apparent flashlight signal, caught on film, marked the end of the attack. More than 100 shell casings of the sort ejected by AK-47s were later found at the site.
  • 1:51 a.m. Law-enforcement officers arrived, but found everything quiet. Unable to get past the locked fence and seeing nothing suspicious, they left.
  • 3:15 a.m. A PG&E worker arrives to survey the damage.

The damage to the substation took 27 days to repair and cost $15.4 Million. In the substation’s 500kV yard, ten transformers were damaged; In the 230kV yard, seven transformers were damaged; In the 115kV yard, 6 circuit breakers were damaged. It was also claimed that a total of 52,000 gallons of mineral oil (used for cooling) leaked as a result of the bullet strikes.

The damage to the fiber-optic telecommunications infrastructure was repaired within 24 hours. AT&T had six cables cut and needed to install new cables to work around the affected area. LEVEL 3 Communications had one cable cut, which was repaired within 10 hours.

The attack on the substation was so over-the-top, especially given that no long-term damage was inflicted, that it more appropriately should have been an entry in Bruce Schneier’s Movie Plot Threat Contest. The trope “orgy of evidence” comes to mind because the attack was so inconsequential for the level of effort expended. Sure it lightened PG&E’s wallet and provided an opportunity for endless sound bites by consultants and lobbyists touting their employers agendas, but nobody’s lights went out as a result of this attack.

So this brings us back to Houdini’s misdirection. Two events occur, one is over-the-top and will obviously lead in the morning media, the other deals with some cut cables in holes next to railroad tracks – decidedly non-spectacular and non-photogenic.

The thing is is that the Metcalf Transmission Substation is next to railroad tracks. And it happens that the railroads’ right of way is used to run fiber-optic cables. I’m sure you’ve heard of SPRINT, which use to be SP Communications, which was founded by Southern Pacific Railroad way back when. Fiber is why all the big name companies in Silicon Valley want to be as close to the railroad tracks as possible!

If we assume that the real target was the telecommunications infrastructure, how would someone tap some of the most monitored lines in the world?

By causing the fiber cables to be so extensively damaged that new sections have to be pulled to work around the damage. This level of disruption would require that any quality/security scans performed – using optical time domain reflectometers (OTDRs) – be re-calibrated after the repairs. The new cable sections could have been pre-engineered to have clip-on couplers (passive taps) built in that exert “micro bending” (i.e., spatial wavelength displacement). If they are detectable by the OTDR they would probably show up as noise near the repaired areas and be ignored. And the voila! Job done.

The next challenge for the strike team would be getting the output from the couplers to somewhere it could be analyzed. Once it was confirmed that the couplers had not been detected, then another team could move in and install appropriate transmitters or couple them into dark fiber for back-haul to data extraction.

We may never know the who/why of this attack. The over-the-top nature of it suggests that it was corporate sponsored as opposed to sovereign. The Metcalf Substation does have some interesting corporate neighbors, but given the nature of the communications traffic flowing in that right of way just about anyone using or traversing that corridor could have been the target.

TL;DR: The substation was actually a diversion.

But there wasn’t much to give credence to the situation until I saw my inbox this morning. Let me repeat something before we start with the new data:

Once is happenstance, twice is coincidence, three or more times is enemy action. And I don’t believe in coincidence.

Lets start with the first article that hit my inbox, USA today.

The FBI is investigating at least 11 physical attacks on high-capacity Internet cables in California’s San Francisco Bay Area dating back a year, including one early Tuesday morning.

Agents confirm the latest attack disrupted Internet service for businesses and residential customers in and around Sacramento, the state’s capital.

FBI agents declined to specify how significantly the attack affected customers, citing the ongoing investigation. In Tuesday’s attack, someone broke into an underground vault and cut three fiber-optic cables belonging to Colorado-based service providers Level 3 and Zayo.

The attacks date back to at least July 6, 2014, said FBI Special Agent Greg Wuthrich.

(Emphasis mine.) Well that’s interesting, but it doesn’t sound all that interesting. The article does note that the incidents have occurred in remote areas but attempts to play it as merely petty vandalism to delay people from getting their cat videos. (No I’m not making it up, see this line…)

Backup systems help cushion consumers from the worst of the attacks, meaning people may notice slower email or videos not playing, but may not have service completely disrupted, he said.

But repairs are costly and penalties are not stiff enough to deter would-be vandals, Doherty said.

“It’s a terrible social crime that affects thousands and millions of people,” he said.

First you have to catch the vandals to fine them, and if this has nothing to do with vandalizing infrastructure but instead tapping it this is a very serious thing. But certainly those lines will help calm those who don’t know details, have the attention span of a squirrel, and don’t have the memory to correlate other external events that are most likely related.

Now lets flip over to the Wall Street Journal.

The latest attack hit several cables in Livermore, Calif., shortly before 4:30 a.m. Pacific time and hadn’t been repaired as of early Tuesday evening, according to several Internet service providers affected by the outage. Some operators complained that law enforcement activity made it harder for crews to fix the problem.

“It’s very inconvenient in terms of getting up at 4 in the morning,” said Peter Kranz, chief executive of local Internet provider Unwired Ltd.

FBI Special Agent Greg Wuthrich said the agency understood operators’ frustration but needed its investigators to look for evidence before anyone patches up the cuts.

“When some of the first cuts were taking place, the cuts and cables were fixed, and there was no evidence, no anything to look at,” he said. “We just need to have a little bit more time to have our people go in.”

I love the complaints about law enforcement making it difficult to repair the communication lines because they want to inspect and collect evidence. This is a classic case of “repair the problem, investigate no further on root cause.” Please stop digging you could induce panic.

Again the paper plays this off not nearly as serious almost as if it’s just some kids out pranking the world. Then we get to the local paper…

The severed fiber optic cables that disrupted Sacramento-area communications is just one in a series of 11 Bay Area incidents in the past year being investigated by the FBI.

Phone, television and Internet services were disrupted in Auburn and the surrounding areas following three severed cables in Alameda County Tuesday morning, according to the federal agency.

Since July 6, 2014, there have been 11 incidents of vandalism to fiber optic cable networks in the greater San Francisco Bay Area.

FBI Special Agent Greg Wuthrich said at this point it is unclear why the cables are being damaged, but said state and federal law enforcement are coordinating on the investigation.

According to communications provider Wave Broadband, three major fiber optic cables were severed at around 4:20 a.m., causing service outages in Sacramento, Rocklin and Auburn areas.

Wait, it wasn’t just one cable shared by multiple service providers, but three different cables? Additionally as these were related to the backbone and given one of the providers involved you just tapped a decent chunk of the internet. Just what the hell is going on down there. I start searching for more information, including something on the Metcalf substation incident to try to cross reference and discover this:

The Silicon Valley power substation that was attacked by a sniper in April 2013 was hit by thieves early Wednesday morning, according to the Pacific Gas and Electric Company, despite increased security.

The substation, near San Jose, Calif., is the source of energy for thousands of customers, and the idea that it was the target of a well-organized attack, and that it might have been disabled for an extended period, raised anxieties about the possible broader vulnerability of the grid. The attack this week did not involve gunfire, and it did not seem intended to disable the facility.

The date on that “theft” is August 27, 2014. The recent string of attacks on the fibre lines started July 2014. Tell me, if you wanted to inspect the response and repair actions of an attack couldn’t you just easily disguise it as a simple theft? You could get up close and personal and inspect exactly how the substation was repaired and what additional actions were taken to harden the substation.

Look, I’m a big fan of Halon’s razor and I hate conspiracy theories because honestly 99% of them are bullshit. But we have multiple, repeat incidents. There were clues and suspicion of possible nation-state involvement which were dismissed. We have an administration who actively works to diminish the significance of attacks and events that surround us and affect us in deep and profound ways. Additionally we see that there are outside nation states who have taken a keen interest in the United States. Just look at the Office of Personnel Management hack, seriously that is a threat beyond what most realize. Then while all this is going on we have people calling to critically weaken our cyber security infrastructure, in the name of stopping terrorism.

There is someone gathering intelligence, placing equipment in the correct locations, and improving their leverage against us. We’re in a technological cold war and we’re seeing the spill over from the physical side of things. Things are not looking good, safe, or secure, especially with over 18 trillion in national debt. Stay safe and keep your powder dry.

Barron is the owner, editor, and principal author at The Minuteman, a competitive shooter, and staff member for Boomershoot. Even in his free time he’s merging his love and knowledge of computers and technology with his love of firearms.

He has a BS in electrical engineering from Washington State University. Immediately after college he went into work on embedded software and hardware for use in critical infrastructure. This included cryptographic communications equipment as well as command and control devices that were using that communications equipment. Since then he’s worked on just about everything ranging from toys, phones, other critical infrastructure, and even desktop applications. Doing everything from hardware system design, to software architecture, to actually writing software that makes your athletic band do its thing.

This is Just Neat

I’ve known about this for a while but I figured I’d share it since many won’t know about the effort that goes into maintaining transmission lines.

Balls of steel.  That is a maneuver that has a pile of  things that can go wrong at any moment.  Most of those things will likely result in death.  That said it can and is done safely on a regular basis.

For those who don’t understand how attaching a helicopter to a 500 KV line doesn’t result in things failing you just need to remember what matters is a difference in potential.  The helicopter is a floating isolated point.  Since its isolated, its reference to “ground” can be shifted.  In this case as the helicopter approaches they use the wand to tie the helicopter to the same potential.  Once the helicopter at line match, so does everything else on the helicopter, including the people.

Once that connection is broken though the systems start drifting apart again.

And lets not forget if the weather turns there’s now another huge problem since wind can cause the helicopter to collide with the power lines.  The lesson there is a helicopter has 74,000 moving parts, all moving in opposition to each other, each with the common goal to kill you.

Still looks like it would be an interesting job, at least as a pilot anyway.  I have no interest on sitting on top of a 500 KV line, despite my knowledge of how it works.  I also know at 500KV it wouldn’t take much to create the current necessary to kill me should a piece of personal protective equipment fail.

Barron is the owner, editor, and principal author at The Minuteman, a competitive shooter, and staff member for Boomershoot. Even in his free time he’s merging his love and knowledge of computers and technology with his love of firearms.

He has a BS in electrical engineering from Washington State University. Immediately after college he went into work on embedded software and hardware for use in critical infrastructure. This included cryptographic communications equipment as well as command and control devices that were using that communications equipment. Since then he’s worked on just about everything ranging from toys, phones, other critical infrastructure, and even desktop applications. Doing everything from hardware system design, to software architecture, to actually writing software that makes your athletic band do its thing.

Quote of the Day – Anonymous (11/29/2012)

Reverse Reactive Power is a quantity defined by the IEEE and we know it is real and exists because the lights are on.

Anonymous – Discussion during a meeting


[We spent more time arguing about this subject than should have been allowed, the instigator just wouldn’t let it go.  He couldn’t understand the subject at hand and felt that the usage of the words were wrong.  Never mind every last power engineer in the room was saying it was perfectly descriptive of exactly what was required for the specification.

For an idea of how bad it was, I was waiting for the meeting to get physical as the instigator became very aggressive about his issue and ignoring the responses because he didn’t understand it.  The person running the meeting was becoming notably upset at the unnecessary delay to moving forward and the fact he was ignoring the responses.

The person couldn’t grasp the concept of directionality coupled with reactive power.  To him it was imaginary it didn’t exist, current either leads or lags.  He couldn’t understand that while it was imaginary in a mathematical sense, it was real, and it results in things such as heat and other issues.  What really got him though was the idea that power has a direction.

Finally one person restated, yes I said restated, that the IEEE has defined this term explicitly and finally justified his point at the end by noting that the fact lights over our heads were working and so obviously Reverse Reactive Power does exist.

It was about this time I was doing everything I could to suppress laughter.

There are days I both love and hate this job.  I will say I’m saving that quote and will paraphrase it when I find a situation like this again.

We spent lunch working on educating him on all the details of what it was, how it was measured, and why there was a direction involved.  He was a bit better but was still a little confused.  I will say it was probably a good thing I was my father’s son instead of my father in this little incident.  I will say I heard his words in the back of my head, the phrase “You obviously have the intelligence of a banana slug” flashed through my brain in my father’s voice.

Don’t get me wrong, he’s smart and sharp, but he knows jack about power, power systems, and what goes into protecting them.  Truly smart men recognize what they may not know and listen instead of insisting everyone else is wrong.  Especially when the others can mathematically show and prove their point.* -B]

*Geek out moment, Reactive Power is explicitly a mathematical quantity.  It is the complex or imaginary portion of Apparent Power.  Forward and reverse is applied merely to indicate the direction of power flow relative to a location.  For example which direction is power flowing in a power line?  To people in this industry directionality is what makes the world turn.  You need to know if the fault is in front or behind your monitoring equipment.  Well if you’re looking at the output of a generator and you see power flowing into the generator (reverse), that’s not a good thing and very rarely intentional.  Something is going to get very warm if you don’t do something soon.

Barron is the owner, editor, and principal author at The Minuteman, a competitive shooter, and staff member for Boomershoot. Even in his free time he’s merging his love and knowledge of computers and technology with his love of firearms.

He has a BS in electrical engineering from Washington State University. Immediately after college he went into work on embedded software and hardware for use in critical infrastructure. This included cryptographic communications equipment as well as command and control devices that were using that communications equipment. Since then he’s worked on just about everything ranging from toys, phones, other critical infrastructure, and even desktop applications. Doing everything from hardware system design, to software architecture, to actually writing software that makes your athletic band do its thing.

Why Did I choose Engineering?

Because of things like this:

There’s a serious personal reward to projects like that.  I do love how there’s even a trigger mode.

My wife just sat staring in astonishment.  Her dad has a prosthetic from an industrial accident.  While he doesn’t have smart limb, he does have a quite high tech limb that flexes and moves naturally.  Most people don’t even realize he has one unless he’s wearing shorts. 

10 years ago, this type of stuff was still fantasy for the most part.  The amount of processing and input gathering required limited the ability to create something usable.  Moore’s law is a freaking awesome thing.

Barron is the owner, editor, and principal author at The Minuteman, a competitive shooter, and staff member for Boomershoot. Even in his free time he’s merging his love and knowledge of computers and technology with his love of firearms.

He has a BS in electrical engineering from Washington State University. Immediately after college he went into work on embedded software and hardware for use in critical infrastructure. This included cryptographic communications equipment as well as command and control devices that were using that communications equipment. Since then he’s worked on just about everything ranging from toys, phones, other critical infrastructure, and even desktop applications. Doing everything from hardware system design, to software architecture, to actually writing software that makes your athletic band do its thing.

Yeah, That’s Always the Solution

Yes, read that title with a serious sense of sarcasm because unsurprisingly we have the following.

Some officials are calling for the U.S. military to take over the managerial structure of the Long Island Power Authority until power is restored on Long Island, where more than a quarter million homes and businesses are still in the dark after Sandy and a snowstorm.

Because fighting a war is so close to restoring and rebuilding the electric power system?  Don’t get me wrong, I have the utmost respect for the military, but there is nothing to indicate they have the skills or abilities necessary to fix this problem.

Now it must be noted that the Navy does have a few men who actually do know something about the power system and distribution, but they also focus on it being aboard ship, with a smaller system with redundancy designed to survive casualties.  Others would look at the military and say, “Well the military has to supply power to their bases.”  Well even the military is lacking the people and skills to do that now days.

By September 30, 2003, most of the over two thousand utility systems owned and operated by the Military Departments are to be privatized. See DoD Reform Initiative Directive #49. Utility systems include systems: (1) for the generation and supply of electric power; (2) for the supply of natural gas; (3) for the transmission of telecommunications; (4) for the treatment or supply of water; (5) for the collection or treatment of wastewater; and (6) for the generation or supply of steam, hot water, and chilled water.

So what exactly would involving the government in the power restoration process do other than create an additional layer of bureaucratic red tape to go through?

There was a considerable amount of damage and it is very serious and not simple.  Many lines that have been repaired remain out of service because there isn’t enough power feeding in to support the line currently.  Not to mention the fact that there has also been damage to the natural gas system and other areas will not have power restored until the gas problems are fixed.

As I said previously:

So what we have is a bunch of distribution points that were/are full of water, need to be drained, the equipment cleaned, checked, maintained, and replaced possibly in some instances.  All of this must be done before re-energizing that circuit.

That takes time, it doesn’t happen overnight, and given the fact that salt water, metal, and electricity is involved  you better do it right.  If you don’t it will be more likely to fail in the future.

Does it suck being out of power?  Yes it does and anyone who thinks a utility doesn’t care about it’s customers being out of power, specifically a significant amount, doesn’t have a brain between their ears.  Each day service is down is a day of lost revenue.  Figure how many people there are, not to mention commercial customers, and then think about how much they’re loosing overall.

Yet again a group of people are screaming the government will magically solve the problem.  Most of those same people actually don’t have a clue about what’s actually going on.

Barron is the owner, editor, and principal author at The Minuteman, a competitive shooter, and staff member for Boomershoot. Even in his free time he’s merging his love and knowledge of computers and technology with his love of firearms.

He has a BS in electrical engineering from Washington State University. Immediately after college he went into work on embedded software and hardware for use in critical infrastructure. This included cryptographic communications equipment as well as command and control devices that were using that communications equipment. Since then he’s worked on just about everything ranging from toys, phones, other critical infrastructure, and even desktop applications. Doing everything from hardware system design, to software architecture, to actually writing software that makes your athletic band do its thing.

Quote of the Day–Ry Jones (11/05/2012)

Mother Nature is a bitch, and physics isn’t taking your call, either.

Ry JonesMy guess is I will never get to drive on the bridge currently under construction

November 5th, 2012


[It is amazing the number of people who have assured themselves that they are some how exempt from Mother Nature being a bitch to them, or that some how they are exempt from the laws of physics.  The worst part about it is those who do that rarely ever suffer the consequences for their decisions and choices, it’s always someone else who ends up taking the brunt of it.  -B]
Barron is the owner, editor, and principal author at The Minuteman, a competitive shooter, and staff member for Boomershoot. Even in his free time he’s merging his love and knowledge of computers and technology with his love of firearms.

He has a BS in electrical engineering from Washington State University. Immediately after college he went into work on embedded software and hardware for use in critical infrastructure. This included cryptographic communications equipment as well as command and control devices that were using that communications equipment. Since then he’s worked on just about everything ranging from toys, phones, other critical infrastructure, and even desktop applications. Doing everything from hardware system design, to software architecture, to actually writing software that makes your athletic band do its thing.

Someone Once Asked Me…

Why I wanted to save all those parts kits from my EE lab classes.  What parts kits am I talking about exactly?  These, including the others I got from friends who didn’t want them:

IMAG0400

 

You see, I had this ability to know I was going to need them in the future for some sort of awesome project.  Well Joe has been trying to get a project done to gather some info and I finally just said send me the schematic and I’ll build it.  Amazingly when I saw the schematic it all became obvious.  I saw what was being done and how it worked and figured out values that I could actually achieve.  Spent  some time at Radio Shack tonight and picked up parts and now every thing is in a nice clean box.

IMAG0401

At some point I may rebuilt it to use the extra connector that is available for power, currently it is using a 9V battery on the inside.  Currently this device is already on battery number two.  I missed a soldier bridge and didn’t ohm it before hooking it up.  Once I cleaned the bridge I did a full functional test with my ohm meter and it worked exactly as intended.

Now you may be wondering what exactly this little box is for.  I will give you a hint otherwise you’ll have to wait for the video.  The first three terminal blocks will have current flowing through them.  The silver BNC connector on the right is for telemetry.  Remember that this box is supplying the current.  I would give you a schematic but it would become obvious.  Those who already know, don’t spill the beans.  You might also consider who the customer is on this one for an idea of what we’re doing.

I will say off on the side I have 100ft of speaker wire and another 25 feet of coax cable.  Now I just need to put some labels on it and we’ll be done.

I will say when the Radio Shack guy asked me what I was building, I made sure to pay first.  The reception as it usually is though was, “Awesome, take video and send it to me.”

Barron is the owner, editor, and principal author at The Minuteman, a competitive shooter, and staff member for Boomershoot. Even in his free time he’s merging his love and knowledge of computers and technology with his love of firearms.

He has a BS in electrical engineering from Washington State University. Immediately after college he went into work on embedded software and hardware for use in critical infrastructure. This included cryptographic communications equipment as well as command and control devices that were using that communications equipment. Since then he’s worked on just about everything ranging from toys, phones, other critical infrastructure, and even desktop applications. Doing everything from hardware system design, to software architecture, to actually writing software that makes your athletic band do its thing.

Fair Winds and Following Seas Mr. Armstrong

So as many of you probably know Neil Armstrong passed away recently.  While it certainly is a blow, it is by no means unexpected:

I am by no means old enough to remember Neil Armstrong and Buzz Aldrin’s historic walk.  I was however lucky enough to grow up being the son of an engineer.  One of my father’s good friends worked on the Apollo program.  I will say I’m not a big fan of NASA, but I will not allow my dislike of them to taint or color the value and accomplishment of the Apollo program.

Growing up I looked at Apollo and it meant one thing to me, if I can dream it, I can build it.*  Hence my becoming an engineer.

I want to see us return in my life time.  But everyone in that program can take solace in the fact they helped inspire a bunch of people to become engineers.  No just at that time, but continuing forward even today.

So pardon me while I go give a wink at the moon.

Good bye Neil and God Speed.

Scott_Gives_Salute_-_GPN-2000-001114

*It is worth noting what sealed my decision to be an engineer was seeing the development of the Physio Control Life Pack as well as the Automatic External Defibrillator.  The fact I saw something my dad and his friends built from the ground up that at the time only had the potential to save lives and revolutionize everything.  Yup, it did exactly that.  The Apollo program’s impact though was no slouch either.

Barron is the owner, editor, and principal author at The Minuteman, a competitive shooter, and staff member for Boomershoot. Even in his free time he’s merging his love and knowledge of computers and technology with his love of firearms.

He has a BS in electrical engineering from Washington State University. Immediately after college he went into work on embedded software and hardware for use in critical infrastructure. This included cryptographic communications equipment as well as command and control devices that were using that communications equipment. Since then he’s worked on just about everything ranging from toys, phones, other critical infrastructure, and even desktop applications. Doing everything from hardware system design, to software architecture, to actually writing software that makes your athletic band do its thing.