Tuesday, January 26, 2010

Licking sandy lube


We have left the Tiffany Navy poster child alone for much of the last year - so let's get an update from our buddy at DefenseNews, Christopher Cavas,
A fresh set of problems with the long-troubled LPD 17 San Antonio-class amphibious ships has sidelined two of the vessels, led the U.S. Navy and its largest shipbuilder into a passionate game of finger-pointing, and raised questions about Northrop Grumman's ability to deliver quality work and the Navy's ability to carry out proper shipyard oversight.

The larger issues are coming from two core problems discovered aboard the LPD 17s, five of which are in service with four still to come.

Of more immediate importance is a problem that, left untreated, could wreck the four large diesel engines that drive the ships. The problem is not new but, having once thought a solution was at hand, the Navy and Northrop are once again trying to figure out why a fix hasn't been found.

Another issue, affecting all the ships in the class and other ships built at Northrop's Gulf Coast shipyards, could - unless it's fixed - shorten the service lives of all the ships. But how and why that problem arose could drive closer to the competence of Northrop and the Navy's inspectors to properly inspect weld work.

Engineers are trying to figure out how debris - "contaminants" in engineer-speak - is getting into lube oil in the large diesel engines that drive the ships. The contaminants cause excessive wear on bearings that support a crankshaft at the bottom of each engine. If the problem isn't treated, the crankshaft will be thrown out of line and the engine could suffer serious damage or even be wrecked.

The problem isn't new, the Navy said, and showed up about a year ago in the third and fourth ships of the class.

"We thought we had it licked," Jay Stefany, the Navy's program manager for the LPD 17 program, told reporters Jan. 21. "And that's where we were until right before Christmas."
Oh goodness.

Let's get geeky.
That's when the newest ship in the class, the USS New York (LPD 21), reported a bent crankshaft in one of the four diesel engines that drive the ship. Engineers found that the shaft was thrown out of alignment by scratches being made in the inner ring of the nine bearings that support the shaft - scratches that caused enough of a difference in the thickness of the bearings to make the shaft wobble.

The scratches are caused by particles too small to see - much of them between 20 and 40 microns wide, or about .00118 of an inch, according to Stefany.
The good new here is that CFFC is taking this seriously.
Early in December, Adm. John Harvey, commander of Fleet Forces Command, ordered Rear Adm. Michelle Howard, commander of Expeditionary Strike Group Two, to begin a Manual of the Judge Advocate General investigation, or JAGMAN, of the problem. The effort reportedly is being led by NAVSEA's Rear Adm. Tom Eccles, the Navy's chief engineer. The investigation is focused primarily on the San Antonio and not the New York, which has yet to transfer to fleet operational control.
Make sure and listen to Midrats on Sunday for a BIW funny - but this just adds to the joy of the folks from BIW.

Byron, step to the microphone,
please,
"We found a higher-than-expected failure rate on quality of the thickness of the welds," Stefany said. The issue was not that, properly hangared, the welds would soon fail in service. Rather, Stefany said, the welds are "critical for shock survivability and for service life. You need [the thicker weld] dimensions to guarantee that." As a result, he said, a ship designed for a service life of 40 years might only make it to 30.

"It's not as catastrophic [as the lube oil problem] but we're working it," Stefany said. "It's not as in-your-face as the engines are - basically it's just putting more welding material on."
Note to our friends in Millington. I will help you find the BA/NMP for LCDR/CDR/CAPT types for you to recode and increase your number of inspectors. I will even help you find additional funding for graduate school engineering education - just give me the top cover and the mandate.

If you don't want to contract me to do it - drop me a line and I will find a EDO CAPT that will do it gratis.

Give NAVSEA and SUPSHIPS more qualified personnel with the right mandate and they will help fix this. All you need to do is set the right priority.

Clean lube oil is about as fundamental an issue as you can get. All the "
Best Places to Work" awards in the world don't mean squat if you can't even get grit out of your lube and measure your welds right.
UPDATE: I got a heads-up by One Who Would Know to clarify something in the article,
.... Chris Cavas, got Jay wrong on the "bent crankshaft". It is out of round, not "bent".
Not the same, but important. Have an engineer tell you the difference - your eyes may glaze over - but he will be giddy as schoolgirl telling you about it. You'll make him happy. Kind of like asking Byron about welds ....

There is a larger backstory on what is being done to fix the lube issue. Good people doing hard work....they just need the resources to do it right.
UPDATE: More goodies being slipped under the door on welds.
DDGs have a similar challenge, and it happened because the entire QA cadre dropped the ball. Uncertified and barely certified personnel performed weld inspections on P1 and P2 piping. A good QA will catch the problems and fix them as you build the ship. It didn't happen that way in this case.
Bad welds accumulated over time until the ships were nearly completed, then everyone went "Oops!" The challenge is that the problem was of unknown magnitude because it involved piping that was lagged and obstructed from view in many cases. LPD-17 class had significantly more welds, so more problem welds which meant they got most of the attention.
A Technical Review Team inspection plan was set in motion last year and the push to find a work around has been going on for a few months. Weld repairs are being made. ASTM weld criteria is being accepted where it makes sense and MIL SPEC criteria will be required where needed (it involves a short equation using the lengths of the two weld legs). There is a detailed Risk Analysis out for review at the moment.

39 comments:

LT B said...

You're totally out of line CDR.  I mean, the CNO says that diversity is our #1 strategic goal.  Welds, shaft alignment, lube oil grit (thus not as slippery as we'd like) are secondary in the quest to spend the tax dollars the best way for the American people.  And besides, diesel engines are new technology.  How could we expect NAVSEA and Northrup Grumman to get this right in a timely and efficient manner?  BTW, have you seen our new uniforms?  They are grand!  US Navy, embracing the repeal of DADT before it's time, and leading the way in 1960's racial reaction to a non-existant problem.  I wonder how that is working for them. 

Jeff Niedenthal said...

Pplease remember that there were engine redesigns, and that the yard requested a changes be made in the design of the lube oil system to accomodate these changes. The design changes were not approved by NAVSEA. There is a mixed bag of responsibility here.

Big D said...

Why is this design so sensitive to microscopic damage, when ships a century ago at times seemed built out of scrap metal and bailing wire?  Did they try to "optimize" tolerances beyond all reason, or something... and if so, to what end?

xformed said...

I'm glad you're in a uniform, LT B.  You're on target in the first sentence.  It's all about the tone of the command, isn't it?

xformed said...

So, the lack of solid education in America comes home to roost?  Do we have history majors who knew where the engineering classroms where running the show?  I'm not a real engineer, but I had a department, and my men graciously trained me in the ways of large machinery, LO/FO analysis, METCAL and a variety of other things necessary to run a ship.  I had several occassions, thanks to my men, to looks NAVSEA "experts" in the eye and ask questions like "OK, we asked you to come and tell us what caused the hairline fracture in the wingwall of the tank in AUX#!, directly outbaord of the Line Shaft Bearing, so fine that fuel would leak out, but seawater would not and your answer is to shrug your shoulders?"

And then there was the NAVSESS engineer (just a few weeks before my tenture) who had no clue as to why #2B clutch was not meshing, but was slightly canted and mushrooming the clutch cup teeth...maybe the fact the 5 year module sound isolation mounts magically were extended to 10 years of life?  Yeah and how about the guy checking shaft alignment values in the ROH just added a J Factor while we stood there in MER #!?  "What's that?"  "Oh, they just tell us to add that in."  Funny...more clutches failed soon after the yards.

BTW, my men wrote the procedure to repair/replace the C/B assy on SPRU Cans.  The NAVSESS guy said "I don't know, no one has done it!"  So...in Valpariso, Chile, GSMC Wiegman, GSM1 Graham and GSE2 Roarh nuked it out.  Brilliant men, I was lucky to have them as my mentors.

G-man said...

As an airdale have to agree with X (and LT B tho I'm sure somewhere there is a CO getting a top fitrep for his diversity achievements whilst 3 decsk below the $hit is getting ready to hit the fan).  Having had a tour at Navsea as flag aide and seen things like diesels turning one way while reduction gears the other on a certain ship built far away from the coast, and shafts falling out of subs, stern gates off gators, I was amazed at our engineering expertise - or lack thereof.  Non-skid that wouldn't stick, boiler tubes on the FID, evaps that didn't evap, and gosh the Admiral had apoplexy on a regular basis.  We hit Long Beach on one west coast swing and hit the Missouri in dry dock.  They had just opened up the reduction gears and those things looked brand spanking new.  I was IMPRESSED by the quality, attention to detail, and meticulous care that those builders put into her.  Where has that pride gone?  This sounds like Detroit of 25 years ago before they let Toyota/Honda eat their lunch.  Maybe we can buy shafts from the French and get the Aussies to come weld for us.  Outsource the minutiae while we worry about uniforms and racial quotas.

Byron Audler said...

All arguable...except the diesels on LPD-17 are German design (made in USA by a company who's main corporate office is on the other side of the Atlantic. I think the problem is the design of the filtering system. Sort of like you can have the most rugged, dependable engine in the world, a helluva transmission and a rear end that more than 20% of max load will blow the third member out from under the car.

Welds: Probable causes:

1) Lack of deckplate observation by NAVSEA inspectors. The downstream of this lack of supervision is supervisors and upper management being mindful of their bonuses and looking at saved manhours and material. Not saying this is what happened, but in nearly 40 years of doing this for a living, if I haven't made every mistake there is to make, I've at least seen them.

2) Inadequate design (an engineer in an air conditioned office deciding that this weld size is sufficient.

Somewhere therein, lies the truth.

Combat Wombat said...

Two words for ya- Merchant Marine Reserve 8-)

Real injunears, who go to sea for a living, and who don't care if it's steam, diesel, or sail-that's diversity for ya!

Put some senior 1675's in these billets- guys who live every day with the results of a shipyard messup, and gut NAVSEA of the "diversity" billets for those who havent had 11 mos underway-across their careers.

Grandpa Bluewater said...

Who is diving and closing out the lube oil sumps these days?  Who is carefully hanging red tags on the air intakes and ducting after checking their sealing from the shipyard environment? Who is pulling samples and paying for analysis of lube oil prior to accepting lube oil, and ensuring sheltered storage for the drums prior to issue? A thousand little checks, protections, inspections; every day, everywhere in the navy. Not so much, lately.

The kids don't know what they were never taught. Instead they are taught political correctness in lieu of the value of plainly speaking the plain truth. Hrmph.

"Integrity is doing well what others may not see."  Old Navy poster.

Byron Audler said...

Be mindful of words like "shipyard messup". My educated guess is that this a problem with design. Given that the Navy no longer has a dedicated group of marine architects and engineers, instead relying on the mil-corp to provide all of that with vetting by the Navy...

It's a situation of getting exactly what you ask for. Seen it, done it, told 'em about it, got ignored, ended up doing it any way...

LT B said...

Grandpa Bluewater,
   I love the Integrity statement.  Good stuff.  maybe we can hang that all aorund the Washington Navy Yard. 

AW1 Tim said...

Sooo... what's up with welding these days?

Seems one of the welders at GE is being investigated for poor or improper welds on a Virginia class submarine. See here:

http://www.defensenews.com/story.php?i=4461573&c=AME&s=SEA

This is the SECOND time this situation has arisen, the prior event being a QA inspector who just gun-decked his job and didn't actually inspect anything for several months.

One back story to this situation is that some in the Submarine business are pointing fingers at Northrop Grumman for pushing the weld story. This last welding problem occured over 2 months ago, and didn't involve any "Subsafe" joints. So why brin it out now?

Seems NG has it's own problems and is likely trying to divert attention from them:

From Molten Eagle, Jan 16th  2010. click the link for more.

http://aquilinefocus.blogspot.com/

"In 2007, <span>the US Navy found</span> that Northrop Grumman employees had used incorrect metal to fuse together pipes and joints on submarines under construction which could have led to cracking and leaks. In 2009, the Navy and the company found that bolts and fasteners in weapons-handling systems on four Navy submarines, including New Mexico (SSN-779), North Carolina (SSN-777), Missouri (SSN-780), and California (SSN-781)', were installed incorrectly, delaying the launching of the ships while the problems were corrected.
The focus on quality-control processes at Electric Boat is intended to distract the spotlight that's been on Northrop Grumman over the last two years for more serious issues. This enables Northrup Grumman to win undeserved USN contracts. answer: Politics.

Look who first published this non-story: Newport News Daily Press. Yes, the home of Northrop Grumman Shipbuilding Newport News."


So it seems our shipmates in the Silent Service are also having problems with QA.

Add to this the problems with the P-8 MPA program being delayed, funding cut, etc, and it points to a systemic, Navy-wide problem in development, construction, program oversight, etc.

I could go on, but you get the drift.

Grandpa Bluewater said...

Big D:

Diesel engines have always been highly sensitive to close tolerances in bearings. Nothing new.

The choice of bearing type brings different vulnerabilities. One material fails first due to heat, another due to pounding of the shaft on the bearing in the space between them, another has little resistance to abrasion. etc. Failure modes, as usual, interact, one leading to another, this is called "the chain of causality". See also "For Want of a Nail", a poem little taught these days.

 Tolerances are close for diesels and and vibration high. When bearings fail (aluminum alloy is vulnerable to abrasion and sterling in other characteristics for bearings, for example) , vibration and etc increase and follow on engine damage quickly, well, follows, to the point of requiring major component or complete engine replacement.  Big marine propulsion diesels don't come cheap, nor do repair parts for same.

Thus isolating the lube oil system and the engine air intake system from the fine particulate dirt and sand blast grit which may be everywhere in a shipyard is important. As is using lube that is clean and uncontaminated in the first place.  Ship and shipyard must partner effectively to get these protective measures done right; housekeeping and attention to detail, system knowledge and forehandedness must fanatically be upheld, lest catastrophe follow. Hmmm, just like on ships, right?

The reason we know this is LT Nimitz, the diesel expert in the 1920's, wrote it all down for us. Yes, that Nimitz.  And yes, Virginia, sailors and diesel engines have been ruining each others' lives on about a ten to 20 year cycle ever since. Because "the burnt fool's bandaged finger keeps wabbling back to the fire" and diesels "neither love nor pity nor forgive" (Kipling again; "Gods of the Copybook Headings" and "The Secret of the Machines", respectively. A dead white poet, imperialist racist, too (if you don't understand what he's saying), hence little taught and less heeded).

Another term, "lack of good engineering practice", is equally applicable for shipyard work force and ship's force personnel.  Or so a cynical old man suspects.

bullnav said...

Aluminum bearings don't do so great when it comes to embedability (i.e., the ability of a foreign particle to embed itself into the bearing surface and stay there instead of ripping through the bearing and leaving a "hole" that will lead to bearing fatigue), but they will do well as far as seizure resistance.  The HD diesel engines for use in trucks typically use a tri-metal bearing which will give you the tight clearances you need but at the same time give you the low-speed seizure resistance you need, which is critical for a diesel developing massive amounts of torque at low engine speeds. 

And all jokes aside about bent cranks, a bent crankshaft is indicative of another failure in the engine, such as a hydro-locked cylinder.  You will typically see bent connecting rods to go along with the bent crankshaft.  Yes, it is catastrophic.   An out of round condition (or excessive runout) may not cause a failure in the short term, but it will cause excessive bearing wear and may lead to oiling issues...

AW1 Tim said...

Indeed, one industry up here in Maine that still has some following is the production of "Oil-Soaked wooden bearings. The advatage was that the oil-soaked hardwood could absorb a fair amount of grit without any noticable wear on the metal parts. These were used in high-contaminent areas, such as machine shops, lumber mills, etc.

An interesting solution to the problem, though.

Byron Audler said...

I wonder if that super-dooper-will-last-till-the-bottom-rusts-out titanium firemain piping is part of the problem? If so, look for blood to wash out the scuppers, and the first lieutenant to start tearing his hair out over the stains.  Dat ain't gonna be a cheap fix, no!

Byron Audler said...

Granpa, during construction, all plate and shapes should come to the ship already blasted and primed. My suspect material would be grinding grit internal to the space (from hull joints inside the machinery space).

leesea said...

go over the Tim Colton an true maritime industry analyst and commentator and see his plan for what to do with No Good Ship Builder NGSB at this link:  http://www.coltoncompany.com/

Calling in MIRAGS US Merchant Marine Naval Reservists might help along with some EDOs?  How about going over to MSC whose professional mariners have been opreating big diesels for decades.  I believe the Navy is now using MSC's lube oil analysis program, maybe the Navy ChEng could use so CIVMAR experience?

Byron Audler said...

Tim Colton is a pretty smart guy. He's been sticking fingers in eyes for a while now.

leesea said...

I couldn't help myself ?~~  Large marine diesels have been used for many decades.  I had them on my Newport class LST.  Merchant mariners have been operating them on Navy owned ships for as long as they have been made.  There is nothing NEW about marine diesels!

My guess having brought TWO diesel powered two ships into service from Avondale is - its NGSB and all that is associated with their crummy construction process (boy that broad sweep will set some off!?)

SCOTTtheBADGER said...

What has the world come to, when GRUMMAN builds crap?  Leroy must be spinning in his grave.

ShawnP said...

Is anyone surprised by the lack of quality in our Gulf Coast Shipyards?

Byron Audler said...

It's come to my attention that NGSB instituted a drug policy after Katrina that involves the hair test. Almost impossible to beat. Might explain why a lot of "experienced personnel" left.

pk said...

big d:

a century ago ships had to light off at least 4 hours before sailing and if they ran at flank speed for a couple of hours they had to go home and "rebuild the engines" which was inactuality popping the bearing caps, drawfiling the parting faces of the bearings and then scraping the bearings to the journal. if this had been done a couple of times they had to shim the bearings to realign to the reduction gearing.......... 

now days on the gas turbines they light off engines about 3 minutes before taking in the lines and leaving. if you doubt me simply stand on the pier and listen when one goes out its pretty obvious.  and yes if we look in the clouds when this happens we see a smile and hear the beaureau of propulsion gang saying on december 8th 1941, NEVER AGAIN!. THIS ABILITY IS A DIRECT RESULT OF THAT OCCURANCE!!!!!!.

i notice that someone drew attention to a fastener scandle. fasteners have a scandle about every thirty years and quite often its the same companies. we used to have dog fights over boiler tubes on about a 15 year cycle but not any more as boilers are out of style. there is another cyclic dog fight and that is with the staves that go into the water bearings in the struts on the propellor shafts. (the rubber or plastic or lignum vitae peels off of the brass or plastic strips that it is stuck onto and we have a multiship multicoast fight over whether the stuff is usable or not)

it all backs up to the fact that quality control MUST speak directly to the big big big dog. all of the others will sweep this stuff under the rug for a really big bunch of ricebowel reasons.

c

pk said...

yeah we never had time/money  to do it right but plenty of time/money to do it over again.

C

pk said...

sir:

in a shaft running in a bearing the shaft tends to "climb" the bearing in the direction of rotation. this makes the axis of load move from vertical to not much more than about 80 degrees in extreme conditions.  local appearant oil pressure is high in this area and low in the trailing area. when the load is released or reversed this area moves back beyond to centerline to the other side of the bearing. thats why in turbines we see the wear on the climbing side as they pretty much run in one direction only. but diesels affect this change from fore to aft then aft to fore once each every revoloution. this is one of the reasons that grit gets pounded into the bearing material rather than being flushed through it. plus the fact that the bearing clearances are just a little bit more that what one of the other folk said the particle size was.

diesel engine cranks, even though they are big, heavy, stout all of those good things, are really about as rigid as a toilet chain. (try putting one in a lathe and indicating it or putting one into a crankshaft grinder and grinding/regrinding journals and throws which are a real bitch to keep straight and true. thats one of the big reasons why diesel crank cases must be RIDGED and machined TRUE.

how about the boys do the traditional fix.

install a LARGE duplex 20 micron filter BEFORE the inlet to the lube oil pumps.

install a DELAVAL purifier BEFORE the 20 micron filter.

clean the purifier every watch (delavals are particularly good in this area) on pain of bread and water for the entire watch that misses it.


don't know about centrifilters as i left this world about the time they were hitting the fleet.

C

pk said...

loss of the Thresher was attributed to gun decked ultrasonic testing of a silver soldered copper nickel joint which leaked into a starter cabinet which scrammed the reactor which ...........

C

pk said...

missouri had open reduction gears because long beach was trying to find out why one of the reduction gear sets were vibrating above ~32 kts. every time she went out and went above that speed she wiped a couple of bearings in the reduction gear and we replaced them. never did find out what the problem was even aftr 5 iterations. i think they finally put a speed plate over the ahead throttle for that engine that said DO NOT EXCEED 293 RPM UNDER ANY CIRCUMSTANCES.

c

pk said...

BULL NAV:

there is an occurance in bearings in turbines and line shafts where the BABBIT developes vertical cracks which turn at a 90 degree angle after a sufficent depth and meet. once they meet a chunk of babbit leaves the scene, probably in tiny particles. it doesn't happen very often and the operators only notice that the bearing temperature goes up for a while but does not exceed the 50 degree for a half hour rule. bearing temperature goes down but never returns to what it was. this condition continues and the journal gets a reputation for running a bit warm.

when the crabs pull the bearing there is a hole in the babit that you can drop a fifty cent piece into. the college boys say that we should make new bearings with thinner babbit layers thereby avoiding the stress that causes this (which usually caused failure within only a few hours of operation) but we used to bore the shells out .250-.375 oversized to get more babbit into them and the things ran smoothly.

the major reason for the thinner babbit was reduced cost of the metal in the rebabbiting process.

C   

AW1 Tim said...

According to the Master Chief, it take 72 seconds to light off a turbine on a Perry or Burke, and be ready to get underway. It's what he did for his 20+ years, so I'll take him at his word.  :)

Grandpa Bluewater said...

PK: Thanks for the nice post elaborating and explaining my points. Since it's been over 3 decades since my MPA days, I'll take all the elaboration and explanation I can get.  Your recommendations on steps best taken on start up or correction of problems noted early on were a nice addition to my blathering on about the application of the law of the 7 P's to new built ships' lube oil systems. I found the discussion on direct reversible engines most interesting, since my experience is with DG applications, which are unidirectional in rotation.

AW1 Tim said...

Yup... it takes only one instance to call the whole system into question.

leesea said...

Hold on their I was speaking about NGSB only!  There are good Gulf Coast yards.  I worked on three from Halter Marine which were pretty good.  I have no direct experience with Ingalls but lots of second hand horror stories

Big D said...

Thanks--that's exactly the kind of answer I was looking for.

Stupid question; if we go all-electric, how much do we gain by getting rid of entire systems like this?  And how do the totally new electrical risks compare?

Byron Audler said...

Well, except for the time to warm up the lube oil, and get the turbines warmed up (which compared to boilers was a very short time)

Byron Audler said...

You'll still need something to generate the electricity... electric drive means no shafts or reduction gears.

pk said...

for those of you that can actually screw a lefthnded nut on a righthanded bolt:

if this guy steffany is right they are circulating the equivelent of 800 grit lapping coumpound through their lube oil system.

C

pk said...

overall machine effeciency will be below the steamers.

they may pick up some effeciency by running the bejammers out of one turbine without having to drag one of the shafts.

c

Marinesurveyor said...

Check out the article in the November 22nd Navy Times: "Civilians to join amphib crew, test hybrid manning"