Railway Investigation Report R98D0184

On 15 December 1998, VIA Rail Canada Inc. (VIA) train No. 600, coming from the Canadian National (CN) Saint-Laurent Subdivision, went through the crossover switches at Ballantyne, Quebec (Mile 8.9 of CN's Montreal Subdivision), to go onto the south track. The CN rail traffic controller (RTC) had programmed his rail traffic control computer to give permissive signals to VIA train No. 60, allowing it to follow train No. 600 after the latter had cleared the next block. On CN's Montreal Subdivision, train movements are governed by the Centralized Traffic Control System (CTC). As soon as train No. 600 cleared the Ballantyne controlled location, the RTC noticed that the CTC had set a permissive indication. He immediately warned the chief controller and the control centre Signals and Communications (S&C) coordinator that a permissive signal had been set even though a train was still in the next block.

Ce rapport est également disponible en français.

On 15 December 1998, as VIA train No. 600 was approaching Ballantyne, the RTC in charge of the Montreal Subdivision programmed his rail traffic control computer to have it set the switches to their normal position so that signal 24R would give a permissive signal to train No. 60 which was approaching Dorval (Mile 11.6 of the CN Montreal Subdivision) behind train No. 600. At about 1659 eastern standard time (EST)^{Footnote 1}, train No. 600 had just cleared the Ballantyne controlled location when the CTC returned the switches on the south track to the normal position. The system then requested signal 24R to display a permissive indication. The system immediately gave train No. 60 a clear to stop signal indication, even though train No. 600 was still in the next block. Pursuant to the Canadian Rail Operating Rules (CROR), a "clear to stop" signal means "proceed, preparing to stop at next signal." When a VIA train receives this signal indication at Ballantyne, the maximum allowable speed is 95 mph. Normally, a restrictive signal is set by the CTC to prevent a train from entering an occupied block at a speed that could cause a collision.

When the RTC noticed that the signal showed that the track was clear, he called the chief controller and the control centre S&C coordinator to warn them of the abnormal situation. They came to the RTC's office to discuss the incident. On his arrival to the RTC's office, the control centre S&C coordinator watched what happened as train No. 60 passed through Ballantyne. He asked the RTC to try to reproduce the sequence of events that had occurred behind train No. 600. Signal 24R gave a permissive signal as soon as train No. 60 had cleared the Ballantyne controlled location. The control centre S&C coordinator decided to take the south track out of service until signal 24R could be checked. He also verified that no other tracks were affected by similar problems.

S&C employees went to the Ballantyne main bungalow and found that a power wire was connected to the wrong relay, allowing the system to give less restrictive signals than it should have. They reported that only signal 24R was affected by this wiring error.

The wire had been installed on 17 November 1998 during extensive work on the Ballantyne main and satellite bungalows. During this time, two projects were ongoing. The first project had begun on 13 July 1998 consisting of relocating three junction boxes in a bungalow and replacing all wires to the external equipment to allow the relocation of the switch control relays in the east satellite bungalow at Ballantyne. The second project began on 18 November 1998 and consisted of allowing easier access to Taschereau Yard through tracks DX1 and DX2. The forecasted in-service date was 23 November 1998 before completion of the first project.

The S&C supervisor had been given diagrams to install a new wire, which was to run from a power source through relay 47LTR (contact 6F) to terminal 12F on relay 27RWCR, setting up a closed circuit. He had assessed the difficulty of the job and concluded that it was a minor task without serious technical implications. He therefore assigned it to one of the S&C installation coordinators. The latter was given the diagrams showing the changes, looked at them and decided on the installation method and testing procedure. In deciding on the testing procedure, he was guided by one of the company's priorities--minimizing train delays. He then obtained his supervisor's approval for the installation method and testing procedure.

The S&C installation coordinator was assisted by an apprentice in installing the wire. The coordinator stood in front of the relay rack with the diagrams while the apprentice lay on the ground behind the rack to make the connections. The diagrams used for this type of installation do not always show the entire circuit on a single page. The S&C installation coordinator must constantly look at several pages to see all the wires to be connected to a relay or to determine the implications of a given connection for the whole system. Relay diagrams are included in bungalow plans, but especially in the case of older bungalows, plans are not up to date. The Ballantyne main bungalow is lit by five bulbs, and a flashlight and spotlight are provided. When employees carry out installations, they use portable neon lamps for additional lighting. There is a small engraved metal plaque or a paper sticker on the back of each relay showing its number. Some relays in the Ballantyne main bungalow were not so identified. Relay 27RWCR was on the bottom row of the rack, and a bundle of wires obstructed the metal identification plaque. The relays are arranged in rows with very little spacing (see Figure 1).

Image

During the installation, the S&C installation coordinator had given instructions to the apprentice concerning the location of the wire, but he could not remember whether he had identified the relay by name or by its position on the rack. The apprentice had made the connection at the back of the rack and had connected the wire to terminal 12F on relay 27NWCR, the one right next to 27RWCR, to which it should have been connected (see Figures 2 and 3).

Image

Image

In order to check the installation, the S&C installation coordinator used a voltmeter to verify that there was current throughout the circuit. He used this method because it did not require the plant to be shut down and did not result in train delays. As it was the end of the work day, the S&C installation coordinator's workload was heavy and these tests are not always considered important, the S&C installation coordinator and his apprentice left the bungalow without performing any other tests. The method used to install the wire and test it did not follow either of the two approved methods (A or B). The S&C supervisor, or a replacement in his absence, did not check the installation to verify the quality of the work and no other test was contemplated or performed to ensure the integrity of the system.

CN has a Quality Control Section which helps employees who do installation or maintenance work to decide how to carry out modifications, make installations, and draft documentation on operating tests or review testing proposals from the Installations and Maintenance sections. The Quality Control Section also conducts certain system tests according to a set schedule.

CN's S&C has a procedures manual to ensure that signal system installations and tests are standardized. The manual, entitled Signals & Communications General Instructions, details the procedures to be followed for many tasks and the tests required to ensure that the system is safe.

The purpose of CN's S&C is to ensure "the installation or revision of S&C signal systems is carried out in accordance with approved plans and will not compromise the safety and reliability of the signal system."^{Footnote 2}

General Instruction (GI) 301(b), Testing New or Revised Circuits and Apparatus, explains the methodology developed by CN's S&C and requires employees:

To ensure that new installations, revisions and replacements are tested in a logical, systematic manner and to provide assurance that the apparatus or systems involved function safely and reliably as intended prior to being placed in service.

GI-301(d) deals with apparatus and circuit revisions and states that revisions to circuits and apparatus must be carried out in accordance with instructions. To provide an adequate level of protection of signal system operation and affected traffic, all work must be done only as assigned and directed by the responsible supervisory officer. The GIs also give procedures for verbal confirmation and repetition when installing wiring.

In view of the physical complexity of some vital signal circuit revisions and their impact on train movements, it may be impractical to make all wiring and apparatus changes at once. This GI gives two approved methods, the best suited of which will depend on local conditions.

1. All preparatory work on wiring and apparatus is performed so as not to interfere with existing live circuitry. Actual revisions are completed during cutover when the system or portion thereof is removed from service in the final stage before operational testing.
2. Most preparatory wiring and apparatus revisions are performed in a manner that allows live circuitry to be opened or disconnected one wire at a time to ensure the integrity of the circuit has not been compromised. Wiring designated to be removed is repositioned outside the equipment racks where it can quickly be discarded. The new wiring is set into a permanent position to facilitate quick cutover when the system or affected portion is returned to service. It is imperative that maximum control over this process be established so the work will be performed in the safest and most efficient manner possible, to safeguard the integrity of the vital signal system through all stages of revision.

The table below lists the authority and conditions required to implement the method used to perform the revision.

GI-301(b) reads as follows:

Before a new installation, change to an existing installation, or replacement wiring or apparatus is placed in service, a complete check and test of the circuits and mechanical features shall be made in accordance with rules and instructions to ensure that the signal system functions as intended.

There are six testing stages for signal circuits and apparatus, as follows:

1. component check;
2. wire continuity check;
3. circuit breakdown;
4. terminal, connection and contact check;
5. final wire check; and
6. operational test.

The three stages relevant to this incident are wire continuity check, final wire check and operational test. GI-301(b) specifies that, in checking wire continuity belonging to an energy loop or string (as in this case), the following three-step procedure should be followed:

• disconnect the end wires of the loop or string from the energy bus;
• check continuity from one of the end wires to each contact and to the other end wire; and
• check no continuity to other energy buses.

When conducting final wire checks, all terminals and contacts on all components must be verified to ensure that they have the correct number of wires and that the wiring corresponds to the circuit plan. An operating test must be performed before a signal system is placed in or returned to service.

In order to ensure that apparatus is installed and tested in accordance with approved methods and documented for future reference, GI-301(e) stipulates that documents pertaining to operational tests and final tests be written in accordance with one of the following arrangements:

1. S&C Quality Control prepares the operational test documents and conducts the in-service tests; or
2. S&C Quality Control prepares the operational test documents and S&C Installations or S&C Maintenance conducts the in-service tests; or
3. S&C Installations or S&C Maintenance prepares the operational test documents and conducts the in-services tests. The test documents shall be submitted to S&C Quality Control sufficiently in advance of "install day" to permit adequate review of the proposed tests.

The manager S&C Installations, manager S&C Quality Control and district engineer jointly decide how to proceed, based on the complexity of the plant and/or alterations, and the availability of resources. In cases of dispute, the manager S&C Line Operations will decide. In the case under consideration, S&C Installations had not filed the relevant test documents with S&C Quality Control.

GI-301(f) is intended to ensure that all signal system plans assigned to specific locations contain accurate, up-to-date information and details pertaining to circuits and apparatus in order to permit the proper maintenance, troubleshooting and testing of those systems. The instruction stipulates that the copy of the plans to be left at the work location (bungalow) must be updated to reflect the work done as soon as in-service testing is completed and the installation is put back in service. It also states that other copies (for the signal design office, central plan office and S&C maintenance supervisor) must be updated to reflect the work done and be forwarded immediately following the completion of in-service testing and in no case exceeding 30 days. On receiving plans, the signal design office must return all required final in-service plans to the central plan office and/or the S&C maintenance supervisor within six months of receipt of the duly amended plans. Since the work in this case had not been completed at the time of the incident, the plans had not yet been updated and distributed.

For the work that had to be done in this case, the copies of the plans which should have been sent to the signal design office within 30 days of completion of the job were sent in March 1999, even though the work had been finished in November 1998. In the interim, there were several copies of the plans in the Ballantyne bungalow since several jobs had been done in the last year.

The mandatory training sessions for all S&C employees include in-depth study of GI-301. All employees must learn the requirements of the manual and must complete exercises and write exams.

Transport Canada's (TC) Rail Safety Branch is responsible for rail safety and conducts inspections and audits of federally regulated railway companies. TC ensures that railway companies install and maintain signal systems that meet the standards of the American Railway Engineering and Maintenance-of-Way Association (AREMA). Some companies send a copy of their procedures manual to TC for information purposes (in CN's case, the GI manual). On receiving a company's procedures manual, TC does not check for compliance with AREMA standards unless an inspector reports a problem in the course of an inspection. In such a case, the inspector finds the relevant section in the company's procedures manual and ensures that it is consistent with AREMA standards.

Railway companies are not required to inform TC when installations or revisions are carried out in a bungalow (as at Ballantyne) or to submit plans of the work. TC inspectors do not routinely check installations or modifications in a bungalow. According to TC's signal system inspection program, 2.5 per cent of all train control systems (CTC, automatic block signal system), interlocked railway crossings at grade, drawbridges and junctions must be inspected each year. The monitoring program also calls for annual inspections of 5 per cent of all grade crossing signals and hot box detectors.

When the rail traffic control computer in CN's Montreal office is programmed to carry out functions automatically (e.g. setting a permissive signal), it scans the track conditions indicated in the rail traffic control centre and, when the conditions are right (e.g. track not occupied, not protected by RTC), it executes the command. When the command is executed in the rail traffic control centre, conditions at the actual location are analysed by the instrumentation in the bungalow controlling the time at which the function will be executed. In this case, when the rail traffic control computer issued the command, the instrumentation at Ballantyne did not carry out all the checks needed before setting signal 24R to a permissive signal. The protection normally afforded by the CTC prevents trains from entering a block already occupied by another train at a speed where a collision would be inevitable. In this incident, the failure of the Ballantyne system created a real risk of collision because one train could have followed another (which could have been stationary) at a speed of up to 95 mph.

When the RTC noticed that his rail traffic control computer showed that signal 24R had been set for a permissive signal, he weighed the risks for train No. 60 and train No. 600. Given that train No. 60 was still at the Dorval Station, he concluded that there was no risk of collision. Since there were no procedures governing what RTCs should do if they suspect that the signal system is faulty, they have to rely on their own experience in taking steps to ensure safe operation. A set procedure would facilitate and standardize decision making and make the system safer.

The control centre S&C coordinator decided to put the south track out of service until signal 24R could be verified. At that moment, there was no way of knowing whether signal 24R was the only one so affected. Without an on-site check at Ballantyne, the integrity of the entire controlled area remained uncertain. There were no procedures to tell the control centre S&C coordinator what steps to take or which tracks to take out of service to maintain system integrity and traffic safety. He had to rely on his own experience in making such a decision. A system for responding to pre-defined conditions based on risk assessment would help maintain the integrity and safety of the system when decisions have to be made concerning any part of the system suspected of malfunctioning.

The procedures manual for ensuring uniform installation and testing of signal systems gives an exhaustive methodology for various tasks. It was drafted by headquarters staff as a book of GIs applicable at all times. Staff at headquarters were adamant that the employees responsible for various tasks were carrying them out according to the manual since training courses explained the rules to be followed. Yet, some employees took the GIs to be merely guidelines, preferring to rely on their own experience in carrying out such tasks. This divergence of opinion shows that CN was not ensuring that approved methods for maintaining uniformity and safety were used.

Ergonomics of Bungalow Installations

Given that employees have to lie on the ground and try to read a small metal plaque with the engraved relay number through a bundle of wires to install a wire to relay 27RWCR and that some relays are not identified, it is difficult to be sure that the installation has been correctly done. The identification of the two relays in this case differed by only one letter (27NWCR and 27RWCR), which increases the potential for error. The S&C installation coordinator and his apprentice did not use standard terminology when installing the wire ("second left" instead of "relay 27RWCR"), and the S&C installation coordinator stood on one side of the rack while the apprentice worked on the other, making it difficult to check the work done.

The GIs provide mechanisms for helping to ensure safety and uniformity. The following sections have a direct bearing on this case:

Performing Vital Circuit Revisions

When modifying vital circuits, one of the two methods approved by the company must be used. Method A is the safer because, since the plant is taken out of service, there is no danger of interfering with signals for trains and there is less pressure to complete the job quickly. When installing the wire at Ballantyne, the involved employees decided not to use Method A because, had they done so, they would have had to take tracks out of service, causing delays to trains for a job that was deemed minor.

Method B is also a very safe method in that several people are involved in several stages of the revision, and authority to use this method must be granted by the project manager and the manager of Quality Control before work can begin. However, when the wire was installed, Method B was not strictly adhered to. The S&C supervisor had not been given authority to use this method, and the Quality Control Section had not been consulted.

The procedure used to install the wire did not follow either approved method and certain established procedures for ensuring safety and system integrity were not followed, creating a real risk of error in installation. Among these procedures is the system of verbal instructions to be repeated by the apprentice. Adherence to this procedure enables two employees to be fully aware of what is to be done, thus minimizing the risk of error in connecting wires.

Signal Installation and Testing Documentation

When proceeding to modify signal apparatus, the manager S&C Installations, manager S&C Quality Control and district engineer have to decide which option to follow in conducting operational testing. In this case, none of these people was consulted because the S&C installation coordinator decided on the testing procedure with the approval of his supervisor. The approved operational testing options involve all relevant parties and ensure that tests are adequate to detect any installation errors before service is restored, thus maintaining the integrity and safety of the system. If none of the three approved options is followed, there is a risk that testing will be incomplete or will fail to detect an error.

Testing

The job in question was to install a new wire as part of an energy loop. The company-approved method for testing wires for continuity involves isolating the loop to ensure that the other loops are isolated. To test the wire that had just been installed, the S&C installation coordinator used a voltmeter to check for electrical continuity in the circuit. Using a voltmeter in this way without first checking to see that the loop or bus is isolated from other loops, one can only be sure that power is reaching the relay to which the voltmeter is attached. Since the loop was not isolated and there was another power source reaching relay 27RWCR, this test could not warn that the wire had been connected to the wrong relay. The S&C installation coordinator decided not to isolate the loop he was working on because it could have had repercussions on the rest of the system. Identifying such repercussions is very complicated since all plans with a bearing on the circuit have to be checked. The other method of isolating the loop would have been to cut off the entire location. This method is considered impractical because it entails delaying trains moving through the Montreal Subdivision, and the S&C installation coordinator and his supervisor felt that this was unnecessary given the simplicity of the task.

Electrical continuity testing was not the only method for detecting mistakes made during installation. Another approved method is the final wiring test, whereby the wiring arrangement and the number of wires at each terminal are checked against circuit diagrams. The S&C installation coordinator decided to leave without performing this test for the following reasons:

• he could not rely on the relay diagram because this page is not updated for older bungalows, rendering it useless for final wiring tests;
• the circuit diagrams, which are accurate, are spread over a number of pages, making it very difficult and time consuming for anyone to identify how many wires should be connected to the relay; and
• the supervisory and quality control groups do not check that all operational tests have been performed, hence the importance of the procedures related to these tests are perceived as lesser than they should be.

In fact, given that there is little emphasis on the tests (as wire count), they are perceived as non-essential. It is recognized that employees will be even less likely to perform a task that they see as non-essential, as well as difficult and time consuming, when the task is to be performed at the end of a long work day in uncomfortable conditions, as was the case for the day on which the wire installation was performed. A wire count test would have shown that the number of wires attached to relay 27RWCR did not match the number that there should have been, and the S&C installation coordinator could have seen that this relay was missing a wire.

Once installation is completed, the usual final stage is an operational test. In this case, such testing was not done because the S&C installation coordinator felt that the voltmeter test alone was sufficient. Since the Quality Control Section's mandate is to ensure that all necessary testing is done to maintain system integrity, it should be involved in all testing decisions. Since the Quality Control Section was not brought into the decision-making process in this case, a defence mechanism was bypassed. CN has no supervisory program requiring that supervisors, or a replacement in his absence, verify the quality of the work done by their subordinates. In this case, the supervisor, or a replacement in his absence, did not go to the site after the work was completed, and a second line of defence against errors was eliminated.

Updating Plans to Match Work Performed

The GI manual specifies that the time lag between completion of the work and dispatch of the plans for that work to the signal design office, central plan office and S&C maintenance supervisor should not exceed 30 days. The manual further indicates that the signal design office must forward the final plans to the central plan office and the S&C maintenance supervisor within six months of receiving the updates. However, these deadlines are not always met. Delay on the part of Installations in forwarding updates to the signal design office holds up delivery of the final plans to the appropriate location (bungalow). In the Ballantyne main bungalow, for example, there were three different copies of plans: a copy of the original final plans and a copy for each of two installations or revisions which had been done. The longer the delay in delivering final plans to a bungalow, the greater the chances that work will be done at that location before revised plans are received. This creates a situation in which several sets of plans have to be reviewed for a simple maintenance job or system modification. It is easier to make a mistake when several sets of plans have to be consulted than when there is only one current plan.

The Signal Design Section does not do all design and updating work, but relies on the services of CANAC International Inc. and firms of professional designers. CN has informed these organizations of the deadlines for submitting final plans, but does not monitor them for compliance. Thus, there may be excessive delays in updating final plans, resulting in several copies of plans in a bungalow for some time.

The Quality Control Section does not verify that all tests have been done in accordance with system integrity requirements for all projects. That section does not test after installation or revision for compliance with plans. Quality assessment of work done by the Installations or Maintenance sections would ensure compliance with company procedures. The immediate supervisor of the employees should also check that the GI manual is followed, and the Quality Control Section could make a note of deficiencies when performing its tests to ensure company-wide uniformity.

TC does not check railway companies' procedures manuals, so there is no systematic evaluation of installation and maintenance procedures to ensure compliance with recognized (AREMA) standards, and there is no systematic inspection of work in progress to ensure compliance with recognized standards, nor observation to assess compliance with documentation on operational testing written or approved by the Quality Control Section.

TC only has a rudimentary national data collection system (IRIS) for its signal system inspection program and it is difficult to evaluate whether trends are developing.

1. A wire was connected to the wrong relay in the Ballantyne main bungalow, so that signal 24R at Ballantyne could display a permissive signal while a train was still in the next block; this created a risk of collision since one train could have followed another (which could have been stationary) at a speed of up to 95 mph.
2. Because of the lack of procedures for RTCs to follow when they suspect a signal system malfunction, RTCs must rely on their own judgment and experience in making decisions to ensure system safety.
3. Without a system for responding to pre-defined conditions based on risk assessment to help ensure system integrity and safety, the control centre S&C coordinator had to draw on his own judgment and experience in making a safety decision.
4. CN did not have an adequate supervisory or quality control program to ensure that approved methods (as in GI manual) were followed in the interests of uniformity and safety, and this allowed Installations and Maintenance staff to do the work without following an approved method.
5. Because of the location of the relays, their proximity to one another and their identification in the Ballantyne main bungalow, the potential for error in installation was increased.
6. Because approved work methods were not followed in installing the wire, the potential for error was greater.
7. Since none of the three approved options was followed for drafting or approving the tests called for after installation or revision of signal apparatus (which involve the S&C Installations and Quality Control managers and the district engineer), the tests risked being insufficiently rigorous to detect errors made during installation or revision.
8. Pressure on employees to minimize delays to trains influenced the S&C installation coordinator's decision as to the tests needed, leading him to deviate from the testing method, which should involve isolating the energy loop he was working on from the others, as approved by the company.
9. Comparison of the number of wires actually connected to terminals with those shown on circuit diagrams was not done after the wire was attached to relay 27NWCR because doing so would have been complicated by the need to review several pages of diagrams to count the number of wires that should be attached to the relay, and because the relay diagram page is not updated. Had such a check been performed, the unconnected wires may have been noticed.
10. Operational testing was not done before service was restored because the S&C installation coordinator thought that the voltmeter test was sufficient. This test alone could not ensure the integrity of the signal system.
11. There may be several copies of plans in a bungalow because of delays in updating the master plan, and all of these plans have to be consulted for even simple maintenance tasks or system revisions. This procedure leaves more room for error than would the prompt updating of a single plan.
12. Without a systematic assessment of installation, maintenance and inspection procedures, Transport Canada cannot ensure that railway companies are in compliance with recognized (AREMA) standards.
13. There is only a rudimentary nationwide database for the signal system inspection program, so that it is difficult for Transport Canada to identify any emerging trends pertaining to installation or testing errors.

Causes and Contributing Factors

A wire was connected to the wrong relay during installation work about a month before the incident, and the method used to test for signal system integrity was inadequate for detecting the error made. The ergonomics of the installation in the bungalow where the work was done and the process for deciding what operational testing was needed contributed to the incident.

Safety Action Taken

The Board notes that, while this incident can be traced to improper signal equipment installation in the bungalow and a field test procedure that can be inadvertently and unknowingly compromised, technical shortfalls and procedural weaknesses are apparent. The verification process used after track signal system projects modification or repair does not always ensure that the system is functioning as intended before being returned to service. Additionally, the quality assurance and regulatory regime do not require verification that work has been completed to specifications. The Board is therefore concerned that the signal system installation, modification and repair processes, as well as the quality assurance and regulatory overview may pose operational safety risk.

The Board also notes that both the RTC and the control centre S&C coordinator observed an abnormality in the CTC signal system and immediately recognized it as a risk to safe train movement. As they did not have specific procedures to guide the decision-making process in such situations, they relied on their experience and judgement to decide on the course of action to take and did not isolate the segment of track concerned. The Board is concerned that the absence of company procedures with respect to signal abnormalities of this type can compromise the safety of train operations.

This report concludes the Transportation Safety Board's investigation into this occurrence. Consequently, the Board, consisting of Chairperson Benoît Bouchard, and members Jonathan Seymour, Charles Simpson, W.A. Tadros and Henry Wright, authorized the release of this report on 16 December 1999.