Sohar Aluminium Substation Sequence of Events Precision Time Protocol (PTP) Enhancement
Problem Overview
Some industrial applications require particularly precise timestamping and synchronisation between multiple industrial controllers, such as motion control and monitoring of electrical systems. SOE modules are a product dedicated to the latter in the Rockwell suite which Philco Systems commissioned in 2008 at Sohar Aluminium’s substation in Oman.
A variety of techniques exist for providing a valid GPS synchronised time source in such systems, which was the component with which the SOE enhancement project was primarily concerned.
These modules received time synchronisation from a series of stand-alone GPS modules, at time of commissioning. This had the drawback that individual GPS module failure could cause time de-synchronisation in event time-stamping between different controllers, particularly when the controllers in question were monitoring common relays or breakers from different points. Once this occurs, a controller will fall back to its internal crystal oscillator for time keeping, which will gradually drift away from true UTC time at a variable rate. It is technically also possible it will actually use the crystal oscillator of some other device on the backplane, but this doesn’t mitigate the drift issue.
Integrated Solution
As a remedy for the discrete time sourcing problem, Sohar Aluminium decided to implement a Precision Time Protocol (PTP) (Wikipedia Editors, n.d.) time server based system. This involved the installation of a Meinberg Lantime server on their network, with compliant switches delivering the PTP protocol to the updated 1756-EN2T modules which are now providing level 2 network access to the Rockwell PLCs. The pre-existing ControlNet RIO was kept in place at the time of commissioning of this project, as the plant is rolling out ethernet upgrades separately, which meant the SOE modules had time synchronisation delivered by means of Coordinated System Time (CST) sync via 1756-SYNC modules, using the relationship between UTC and CST. This leaves some scope for enhancements once the ethernet upgrades are rolled out to this part of the plant.
Wherever possible the design specification of the substation requires high availability implementation of its systems, which means integrated components were all redundant, including the Meinberg Lantime server’s GPS source, although it is worth noting that total loss of time keeping from the server has some contingency in the PTP time source priority system, which allows one of the controllers to take over as a primary time source for the network, meaning that oscillator drift will not affect the comparability of event time stamping even after long periods of downtime.
The opportunity was also taken to increase the resolution of the SOE time-stamping stored to the dedicated historical database from milliseconds to microseconds. This involved some MSSQL table recreation (performed without losing previously created records) and the transfer of Unix Epoch to local time conversion to the PLC, which has the side benefit of ensuring datetime issues are not raised by locale settings on the server, and is viable in this case because the data is not used by the overall plant historian, which is more likely to be cloud connected or used multinationally in some other way,
The commissioning process included server rack installation, PTP switch configuration on a fully redundant network layout, PLC software modifications to propagate the Time Synchronised UTC/CST relationship to the SOE modules and hardware installation of the newer brand of ethernet modules (which were replacing the 1756-ENBTs used at the time of plant commissioning in 2008), modifications to the FactoryTalk Transaction Manager configuration and modification of the MSSQL installation (the two latter items also being redundant on a pair of level 2 data acquisition servers), which took place over the course of 2 weeks on site in Sohar, Oman. As a result of the module replacements, PLC downloads were required, which involved operating on rectifier controllers 1 at a time in order to prevent any critical interruptions to the Potlines.
Outcomes
The true benefits of a system with such precise time stamping of events is that in the event of disaster, fault finding is faster and more reliable. The root cause can be analysed clearly, without aligning exact timing between multiple different controllers, in some cases using assumptions based on electrical design. The higher degree of accuracy in dashboarding is also a nice side-benefit to everyday users of the system, operating from the FTView SE SCADA, who can more precisely audit events during their shifts.
References
Wikipedia Editors. n.d. “Precision Time Protocol.” Precision Time Protocol. https://en.wikipedia.org/wiki/Precision_Time_Protocol.
