Case Studies

Industry: Manufacturing/Industrial

Client: Automobile Manufacturing Global HQ 

Service: Forensic Failure Analysis 

Solution: Discover, Mitigation and Control of Contaminant

Executive summary of the situation

An automobile manufacturer was experiencing ongoing hardware failure in its design center, causing significant downtime and production loss. After repeated efforts to determine the cause, the client contacted WES.net to find a viable solution.  WES.net suggested a forensic and deductive approach in the effort to baseline conditions and determine the root cause of the failures. WES.net assessed the facility from a broad scope, evaluating both internal and external elements affecting computer operations (temperature, humidity, contamination, corrosivity, power quality, grounding.)

 

An approach to solving the problem

The systematic and scientific approach taken by WES.net effectively determined the root cause of system shutdowns.  Hypothesized causes of hardware failure were eliminated through keen observation and the analysis of supporting data. Initially, it was suspected that power quality issues were the main cause of hardware failures. However, further investigation would lead to determining the actual origin of system failures.

 

Overcoming challenges

After eliminating an array of probable causes through data analysis and laboratory results, the source of the failures was determined to be creep corrosion.  The manufacturer’s design team had recently switched to new sulfur-based modeling clay.   When heated, the clay released gaseous particles into the environment.  Combined with elevated humidity levels in the facility, this expedited a chemical reaction occurring within various pieces of IT and computer hardware. Forming a highly conductive pathway between connectors on the circuit boards, this led to the eventual shutdowns they were experiencing.

 

Moving forward

WES.net provided solution-based options rather than a list of services to perform.  The initial recommended involved the modeling team switch to sulfur free clay which would quickly dissipate and subsequently eliminate the potential of further gases being released.   By not disrupting the critical environments, this adjustment would decease further contamination development.  Secondary actions included remediation of the facility to eliminate both on-going and potential long-term effects of sulfur gases and creep corrosion.  Currently, WES.net continues to consult with the automobile manufacturer on further remediation efforts.

 

Industry: Healthcare/Hospital

Client: Medical Facility Disaster Recovery Plan

Service: Critical Facility Assessment

Solution: Critical Infrastructure Management                                       

Executive summary of the situation

Following an intense hurricane season, a renowned hospital and healthcare system became painfully aware of its vulnerabilities following a natural disaster. Torrential rains and winds battered their community, leaving them without power for days rather than hours. As these types of events increase in frequency, intensity and reach, the medical community took a hard look at its’ emergency preparedness and energy contingency plans to avoid future shortcomings.

An approach to finding a solution

Consequences reach beyond structural borders and local economies when medical facility continuity is disrupted.  What happens today can have lasting effects on tomorrow’s services, functionality, and future resiliency. Taking a proactive approach to disaster relief preparedness, the hospital system authorized WES.net to manage a full assessment of their vulnerabilities in order to improve on preparedness efforts for extended power outages and downed services.

Overcoming challenges

Emergency preparedness in healthcare effects the ability to recover rapidly, which is imperative to the care, comfort and overall servicing of patients. Challenges due to power outages and communications loss, gas shortages due to disruption of transportation networks, and loss of supply chain delivery effect speed to the restoration of energy resources.

Long term outages have dire effects when a facility falls off the energy grid.  It was recommended as an immediate course of action the development, deployment, and implementation of de-centralized and interconnected microgrids.  Looking to leverage newer, cleaner and more efficient technologies, they hoped to provide true energy resiliency to better manage usage, reduce costs and increase efficiency.

Moving forward

The hospital system has since partnered with state and federal agencies, local energy research enthusiasts, private interest groups, energy providers and consumers in a focused effort to improve resiliency and continuity of services in the face of disaster.

Industry: Government/Military

Client: Government Facility

Service: Assessment

Solution: Install of Active Corrosion Classification Monitoring System

Executive summary of the situation

A systematic and scientific approach was used to determine the root cause of hardware contamination and subsequent failure in a government facility. Following multiple attempts on their own, WES.net was able to identify creep corrosion at the root of the problem.

An approach to finding a solution

After review and analysis of evidence, laboratory results and site conditions, WES.net concluded that the data center environment showed evidence of sulfur contamination and high levels of moisture, capable of supporting creep corrosion. Although they were able to determine probable cause of hardware failures, the source was complicated to identify. Creep corrosion coupon results showed the environment to be corrosive to a moderate rating, with magnified elevated levels of moisture.  Reacting with copper and silver connectors commonly found on circuit boards, a chemical reaction was created, with a highly conductive pathway between connectors in the hardware.  This led to eventual failure, yet still the source of the sulphur eluded them.

Overcoming challenges

Following further physical inspection, it was noted that after fireman ventilation systems ran, contamination occurred.  Regardless of how the supplemental system was engaged when air conditioners were replaced, the contamination source could also have been created through an adjoining office when doors or windows was opened.  Both paths were considered. Elevated moisture levels caused by lack of maintenance controls meant relative humidity was the only moisture measurement taken by the air conditioners.

Moving forward

An immediate recommendation was to review controls and set regulation thresholds for temperature, humidity and de-humidification controls. Hardware sees a temperature intake at the midpoint of the ASHRAE recommended range to maintain correct relative humidity levels so that the dew point remained within that midpoint range.

WES.net recommended they create negative pressure in adjoining offices so when doors open and close, less air would enter. They would continue to identify the source of the gas while looking to install an active Corrosion Classification Monitoring System that logs activity on a continuous basis.

The BladeRoom System and Methodology are protected by international patents, patents pending and by registered design. BLADEROOM and the “BladeRoom” Logo are trademarks registered to BripCo (UK) Ltd. © Copyright 2017. All rights reserved.

Industry: MSO/Telecom

Client: National Cable Provider  

Service: Facility Management and Consulting

Solution: Install Recommendation of BladeRoomDC Microgrid, Voltserver

Executive summary of the situation

A multi-service provider in the cabling industry was in dire need of changing existing architecture to accommodate a NextGen network.  The business had two major requirements.  The first was to meet a tight implementation schedule with rapid occupancy.  Realizing early in the process they were facing financial and technological limitations, the organization selected modular construction over build to suit for the enhanced data center.  This included an on-site microgrid and a fully operational solar back up system.  The facility would be factory assembled and built with enhanced operating modes.

An approach to finding a solution

WES.net approached the situation using their proven consultative approach, recommending a combination of best of breed technology in pre-engineered facilities with new electrical distribution topology. However, limitations brought forward by the client challenged the initial recommendation made by WES.net, so they immediately constructed a new outline suitable to the clients’ wants and needs.

Overcoming challenges

Upon full review of the scope of work, it was determined that existing current technologies were obsolete with legacy design limitations.  There existed an inability to adapt to emerging technologies or transition to a virtual environment.  The organization was also in need of a rapid recovery contingency plan, with a full grasp of how to monetize improvements in order to expand long term capabilities.

Moving forward

WES.net recommended they project manage the full life cycle of the install, from data center design sign-off to handover of the facility.  The team suggested the integration of power, space, and environment for the creation of an MSO edge facility.

Industry: Government/Military

Client: Data Center in European Headquarters

Service: Gaseous Contamination Assessment

Solution: Facility Management and Intelligence

Executive summary of the situation

WES.net was authorized to conduct an investigative study at a critical hub facility in Europe.  Taking a scientific approach to the effects of unexpected hardware and systems failures, the technical disruptions were determined to be caused by creep corrosion.

 

An approach to finding a solution

Initially, multiple hypothesized causes of contamination were eliminated by supporting observations and data.  Ultimately, the experts from WES.net determined that sulfur gas was the main contributor to contamination, and subsequently, continuous hardware failures.  Additional analysis confirmed the expedited contamination was enhanced by elevated moisture levels in the hub space.  The combination of gas and moisture created a chemical reaction in the hardware, forming of a conductive pathway between connectors.  Additionally, chlorine gas was detected as a contributing factor contributing to system failures. 

Entering the data hub by way of outside air filtration systems, significantly higher levels of copper corrosion was identified at the entry point as compared to the corrosion classification coupon located inside.  Sulphur gas from a stand-by generator circulated into the building from outside air intake.  Additionally, chlorine was entering the building from cooling coils closely situated to the intake system.  Although the coils were jet washed numerous times, the water vapors contributed to intake rather than hinder chlorine exhaust.

Overcoming challenges

Elevated moisture levels created poor operating conditions within the space. Room temperatures continued to rise without proper consideration and alteration.  WES.net made recommendations to avoid further issues by addressing increasing temperatures while striving to maintain a proper dew point level in order to minimize contaminant gas concentrations.

Moving forward

Through proper monitoring of the facility, the business would address the issue of sulfur gas contaminants in the facility. The implementation of an additional chemical filtration system on either the outside plants or a dedicated chemical filtration system would be required.