Auckland's Power Outage or Auckland - Your Y2K Beta Test Site Last updated 24 May 1998 The following writeup is a (hopefully) more balanced view of what's going on than the one being provided through official channels, I'll keep this updated as new information comes to hand. If anyone has comments or updates on this, please send them to peterg@kcbbs.gen.nz, or pgut001@cs.auckland.ac.nz after power is restored. The city of Auckland, with a population of just over a million people, is New Zealands largest city and has its power provided by Mercury Energy, who have four 110kV cables feeding the central business district. Because of one or more of the reasons given below, all the cables have failed, leaving the central city without power since the 20th of February. So far this has affected (at various times) a number of banking data centres (the first day the power went out was on the Thursday when everyones pay is supposed to be processed - the data centres themselves have generators, but the sources feeding them information don't), the stock exchange, some (unidentified) central city post office buildings, customs and immigration, inland revenue, internal affairs, social welfare, the Auckland City Council, the central police station, Aucklands main hospital and medical school complex (they have generators, but one of them failed, leaving the childrens hospital without power for awhile), the city campus of the university and technical institute (affecting 30,000 students in the middle of enrolment), several TV and radio stations, many of the major city hotels, a virtual who's who of national and multinational companies and corporations, and God knows what else (the government departments have tentacles all over the city, so it's not so bad for them). Although many of these places have generators, there were various glitches in switching over and one or two breakdowns which have caused problems, and most of the generators can't handle anywhere near the load being placed on them but were designed to power only essential services. It's possible that the power company may not survive the lawsuits which follow this (taking out some suburb is serious enough, but taking out the central business district with its cluster of multinational accounting and legal firms, banks, government departments, and whatnot is really bad). Mercury have a web site http://www.mercury.co.nz/ which contains updates on the situation somewhere on it, be aware that this site is subject to the usual degree of spin control (there have been discrepancies to date between their statements to the media and what's actually happening, and power-outage related pages on their site tend to appear and disappear at random as it suits them). Effects on the Central City --------------------------- For those who know the area, the affected area of the city is bordered by Quay Street, Victoria Park Market, Grafton Gulley, and the motorway. At first the estimated time to repair was a week, then the official estimates being fed to the media were 1-3 weeks, but the estimates from power company workers are a month at least (the official figures change constantly, they seem to be getting slowly worse). Usually it takes a minimum of 3 weeks to repair a fault in a cable, however with crews working around the clock and the bare minimum of testing of the repaired cables, it may be possible to do it in a week, provided it holds together. However, see the comments further down on overloaded cables and cutting corners - already there are problems with leaks and blockages being discovered in the gas-filled cables which are pushing the repair time out further and further. With political concerns overriding technical issues, it may be that any repaired cables rushed back into service will fail again due to the hurried nature of the work. Workers from other power companies are being brought in and working in civvies with company logos on their equipment painted over to hide the extent of the problem. Workers were flown in from Sydney, Australia to fix the cables because it's too costly to keep specialists like this on hand at all times, so there's noone in NZ who can do this sort of work. The estimate is that it'll take about a week without power to redo these, and if the load placed on them is too high they'll fail again (the faster they bring the cables back online, the greater the chance of them failing, making it necessary to start again from scratch). Normally it takes a week of testing alone after a cable has been repaired, this has been brought down to a single day because of the urgency with which they're required, which isn't a good thing. As it turned out, the two repaired oil-filled cables did indeed fail when they were tested, leaving a stopgap cable link being built along the central cities rail access corridor (more on this below) as the only hope for restoring power. Although Mercury haven't admitted this, it's very likely that the two failed cables are now beyond repair, since they haven't mentioned any further timetables for repair efforts but are banking on getting power in via the stopgap cable (see the comments further down on guessing the real story based on Mercury's statements, which tend to be driven by politically expediency rather than cold realism). The time to lay the stopgap cable was originally quoted by Mercury as taking six weeks, but the figure they're now giving is two months - "If you live or work in the central city, stay away for the next two months". To put this into perspective, Mercury are telling government departments and multinational corporations to shut down their operations in the city for two months. For the moment the city is hanging off a single gas-filled cable nearly half a century old, with hopes of restoring another equally ancient gas-filled cable within the next week. Because of their age and fragility, both will be operating well below their very limited capacity, providing only about 18MW each. Initially (when Mercury were quoting one week to restore power), the mayor told businesses in the central city to either close or relocate for a week or more. Staff were relocated as far away as Australia. Due to the enormous impact the continued outage will have on Auckland, the mayor is now considering declaring a state of emergency (this hadn't been done until now because it wouldn't have had any real effect except to impose another layer of complexity onto an already complex mess). The potential losses if major corporations and businesses relocate out of Auckland are huge, there are already several large organisations who have left and who aren't likely to return (in some cases the contents of multiple floors of large office buildings were moved lock, stock, and barrel to other locations). Wellington (the capital) benefitted enormously from this as companies relocated there from Auckland. In the last five years, Mercury Energy have followed the present economic wisdom of aiming for efficiency and a good return to their shareholders (the Mercury Trust), raised power prices, reduced their field workforce by half, and raised management salaries by 30%, with total revenues of $580M in 1997. In addition for the last three years their energy has been poured mostly into a pointless (and ultimately fruitless) struggle to take over their neighbouring power supplier, Power NZ, which cost Mercury $300m. In the middle of the first week without power, the Auckland City Council called an emergency meeting in the town hall to discuss the problems people were facing. Some of the business owners who attended were on the verge of bankruptcy because of the lack of power, but Mercury didn't even bother turning up, an act which the mayor described as "a disgrace". This sort of thing isn't endearing them to their clients/victims. There seems to be a strong feeling that those who got the huge pay rises and bonuses when things were going well should now take responsibility when things are going badly. One company manager was even more blunt: "fix the problem, make sure it doesn't happen again, then resign". On the afternoon in which Mercury directors held an emergency meeting to discuss the implications of the two repaired cables failing, the meeting had to be moved at the last minute because one of their dissatisfied customers/victims threatened to blow them up. The police are taking the bomb threat seriously. Because the central city was without power, there was a civil defence callout in the first few days to avoid a potential crime wave. Police were called in from other parts of the city to patrol the city center, leading to a lack of policing in other areas. The lack of power is affecting building access control systems and alarms, buildings have to have doors propped open so people can get in or out, so there's no real protection for the building contents. The services of private security firms are in great demand. By the end of the week the policing and security issues had been more or less ironed out, and the civil defence headquarters were closed as they were no longer needed. Since water and sewage rely on electrically-driven pumps to get them into office blocks and towers, these services often aren't available either. What little power is available is being used by emergency and civil services as far as possible, with other services like traffic lights being run if there's anything to spare. Many office blocks have no power, water, or sewage services available. Combine the lack of sanitation with abscence of airconditioning and you can imagine what conditions are like in parts of these buildings. One exasperated inner-city apartment owner summed the situation up with "I spent a million dollars on this apartment and I can't even flush the toilet". There's a serious problem with food in restaurants spoiling due to the lack of refridgeration, with health inspectors out in force checking for compliance with safety regulations. The Ports of Auckland shifted 500 containers to Wellington for unloading before the goods in them spoiled, and the city council parked a large refridgerated container unit powered by a generator in a city square for use by restaurants to store frozen food, although it's unclear how much this will really help. Retailers were told to sell all they could before it spoiled and dump the rest. Provisional tax payments are due at the end of the week, inland revenue have so far indicated that they won't be making allowances for the fact that businesses can't make the payments until power is restored. The matter has now been taken to the minister. Several businesses had gone bankrupt by the end of the first week without power, and the chairman of the Auckland Chamber of Commerce has said that for a number of small businesses, the best option they have is to walk away from their businesses and start again. In one 10-15 story office block, sprinklers were activated by the power outages and continued spraying water into the building for quite some time. A comment from someone who saw the aftermath was "They may as well demolish the building and start again". Generators and Disaster Plans ----------------------------- This sort of outage is (needless to say) extremely unusual. In 1996 there were, as a nationwide average, three power outages (of < 10 minutes duration) a year, with a major outage (1-2 hours) occurring once every two years. This one is really going to screw up the end-of-year balance sheet. One company flew in a generator from Poland to try and keep things running. Another company is flying in generators from Australia. An online news site was contacted by several US companies wanted to buy advertising space, but they were rather disappointed to learn that 110V 60Hz generators weren't going to sell too well in a country which runs off 230V at 50Hz. The lack of power is a UPS vendors dream, they're almost impossible to obtain - one company asked that their order of UPS's be shipped with a full charge. Many of the systems still running are hanging off heirarchical networks of generators feeding large UPS's feeding smaller UPS's feeding computers, routers, and whatnot. One contributor reported seeing 30 cases of newly-arrived UPS's outside a companies offices in the central city. Another contributor reported that a company bought 30 laptops and cellular modems and sent the bill to Mercury. ISP's located outside the central city are telehousing customers from within the affected area. One central-city ISP was saved when one of their customers drove his generator in from the Coromandel (several hours drive from Auckland) and installed it by torchlight on Saturday evening. Other ISP's on the edge of the blackout area have various contingency plans ranging from switching to generators as much as possible to relocating further out into the suburbs. Computerworld magazine is being run from a garage in Newmarket (located a few km from their usual central-city offices), the editor suggested renaming it to Garageworld. Vendors have reported huge jumps in notebook sales as people relocate their offices into homes outside the central city area, cellphone sales have jumped 25% and cellphone rentals 300%, with one large dealer running out of phones entirely. This has lead to a significant increase in utilisation of the cellular network, with the system running almost at capacity in the first few days of the outage. Mercury are busy hiring out what generators they have at $1000/day to the businesses they've cut off, and on the first Wednesday of the outage chartered an Antunov 124 and flew in six large generators scraped together from all over Australia at a cost of $1.5M. The producers of Xena and Hercules have lent the generators they normally use to the city (thanks guys). Over the weeked at the end of the first week, Mercury chartered a Russian Aleutian cargo plane from Singapore and made two more trips to bring in another four Aggreko generators from Brisbane and Sydney in time for the start of the next working week, providing a total of 34MW of generators in operation in the central city by the following week. However no more large generators can be brought in by air in the near future because further aircraft capable of carrying them aren't available in this part of the world - the Russian cargo planes have moved on to other work, and the only option left is to bring them in by ship. Apparently the idea of moving ships from the naval base on the other side of the harbour across to the Auckland waterfront to act as floating generators was considered, but there are problems with feeding the power from the ships to the city. There's also the problem that there's nothing around which can generate even a fraction of the power required. Another idea which was considered is using one of the Cook Straight ferries (which could in theory provide around 10MW) as a floating generator (the term "ferry" is a considerable understatement). Currently a couple of waterfront businesses are being run with power from ships acting as floating generators, and when both repaired cables failed their testing, Mercury finally brought in a diesel-electric trans-Tasman freighter, the Union Rotorua, to act as a 12MW floating generator, and is considering bringing in another ship or installing generators on barges. However the problem mentioned above of feeding power from the waterfront to the city remains, with the Quay St substation only being able to run to a certain limit (again, there's a problem with a single point of failure - if the substation goes, there'll be serious trouble). If a state of emergency is declared, it might be possible to use it to string cables through the streets to other substations and feed in power from there. The cost and overhead of running all these generators (without including the ship(s)) is fairly frightening, Telecom's generators alone are burning up 1,700 litres of diesel an hour (that's about 40 tons of fuel a day just to power the central city phone network). When asked about the diesel tankers parked outside it, one petrol station some distance from the city centre said that it was moving around 40,000 litres of diesel a day. Based on known generator capacity and consumption, a rough estimate puts the central cities usage as high as a million litres of diesel a day (I'm open to corrections on this one). There are concerns about how long the supply of diesel can hold out under these load conditions. Some companies have power through various lucky coincidences, for example Datacom (who provide payroll services) were required by a contract with a client to provide redundant backup systems which were set up only weeks before the power outage. Other organisations were less lucky, with Air New Zealand finding out that although their data centre has enough generating and backup capacity to keep its huge mainframes running, the creeping takeover of LAN's (without any backup power source) meant they had to move their staff to other centres around the country. The organisations which do have generators are finding that, although you can run them for the usual smoke test every few months, they can't supply power for any extended period of time (the generator from Poland couldn't actually be run once it was landed due to the excessive amount of noise it produced, although this problem has now been resolved). Other companies have found that their generators have problems with fuel tanks placed too close to the generator or inadequate cooling and exhaust ducting. On the first full day without power, there were four fire service callouts to generators which caught fire because they weren't used to running for extended durations. There have also been one or two fires caused by incorrectly wired generators and emergency power cabling. There is by now enough raw data about Disaster Recovery Plans which don't work for an entire conference. Lawsuits, Government Enquiries, and Who's Really to Blame --------------------------------------------------------- From talking to people in various affected central city buildings, as soon as the power comes back on the affected law firms will be handling enough lawsuits to keep them in clover for years. In theory the current commercial monopolies inherited the privileged positions of the old Power Boards from which they're descended, making it impossible to sue them for failing to provide a service. The only thing they can be sued for is negligence, there is mounting evidence that this will be possible. Mercury say that the cables were all dug up quite recently to sleeve the joints as required by the Resource Management Act to prevent loss of oil or gas into the environment, and that this indicates that they were being maintained (this, however, seems more like "wave a dead chicken over them to stop us being fined under the Resource Management Act" than any real maintenance). Contradicting this, the chairman of the Mercury board has said that the best thing to do is leave the cables buried and not touch them, so digging them all up may have made things worse. When asked whether he believed Mercury had been negligent, the chairman hedged his bets by stating that they hadn't been negligent in trying to restore the cables. In 1993, five years before the current crisis, Mercury Energy had become sufficiently concerned at the condition of the cables that they asked Leyland Consultants to prepare a report on them, which recommended that the main cables were at the end of their life and should be replaced (the engineers at Mercury had been expressing concern about the state of the cables for some time before that, see below). In fact it was later revealed that Mercury's strategy for the cables was that any failure in the gas-filled cables (which was seen as not too unlikely) would result in their load being switched over to the oil-filled ones which would then carry the full load. This strategy was based on the dodgy state of the gas-filled cables and the relatively good history of the oil-filled ones. When the first cable failed, the fallout from the outages it caused were such that Mercury avoided making any real cuts to try and reduce the load on the remaining cables (this weeks Computerworld, prepared before the outages, contains an article reassuring everyone that the problem is solved and everything is OK). Mercury ran an emergency feed for several miles over a string of poles, which had hardly been completed when the second cable failed. They then tried to force a full load over the remaining cables by management will-power alone, which unfortunately wasn't enough to overcome the basic laws of physics, and everything which was left failed as well. It's not unlikely that the combined legal resources of everyone they've annoyed will find enough material in there to get to them (there are probably armies of lawyers sitting around candles right now scrutinising the relevant legislation), and the case will eventually get heard in the Auckland high court by a judge who's just spent a week working by candlelight because they don't have power either. There are plans for multiple mass class action suits against Mercury, one group has even said they'll get criminal charges pressed against them for crimes against public welfare. I think I'll join the class action suits as well, the fact that the university machines are down means that I've had to use tin to read news for more than a week, that's got to be worth several hundred thousand dollars compensation for mental anguish. The Prime Minister has already made a plea for people not to engage in a witch-hunt against Mercury, but this appears extremely unlikely as things get more and more desperate each day. So far Mercury seem to have blamed the whole mess on an act of God, knowing that it'll be more difficult for people to get compensation cheques out of Him than from Mercury. The government has said they won't be offering any compensation because it's strictly a matter for Mercury and the Auckland City Council, which has upset some Aucklanders because the government seems to be happy to hand out compensation whenever things go wrong elsewhere. Mercury and the Auckland Power Consumers Trust have set up a $2M fund with each contributing $1M to help people and organisations affected by the power outage, although the maximum possible payout of $2000 is of little use to businesses faced with $1000/day charges for generators. There's also concern that the payouts may be a bit of a booby trap, with anyone applying for funding finding it difficult to claim further compensation from Mercury in the future. In any case because of the tiny amount being paid out (an average of $750, which can't even begin to sustain businesses who have lost nearly two weeks worth of revenue) almost noone has applied for funds, and most of the $2M is still sitting there untouched. Consumers were, however, cheered to find out that although there was no money available for them, $500,000 had been made available for an enquiry into why the whole mess had occurred in the first place. A government enquiry into the whole matter was announced fairly early on (the results of this are given further down), one of the things which was supposed to have been looked at was the extremely peculiar relationship Mercury have with the law firm Russell McVeagh. Although Mercury is 100% owned by a trust, of the 9 Mercury directors only 4 are appointed by the trust, with the majority of 5 being appointed through some mechanism which involves Russell McVeagh. Much of the Mercury board appears to consist of lawyers who know nothing about running a power company (as a radio sketch by Fred Dagg, an ex-NZ comedian, put it, "I'm an accountant, that's why I'm running a power company"). Mercury then in turn appointed Russell McVeagh to act as their legal advisors. Although this is just the usual paranoid structure set up when you ask lawyers to manage things for you, to the public it looks very suspect, and was regarded as being a primary target for any government investigation. This corporate structure is extremely nasty for the nominal owners (the public), because although they don't control the company, they do carry all the liability for it. Even if Mercury is successfully sued, the public will just end up paying for it through higher power bills. Noone except the lawyers who set the whole structure up will ever make any money out of this. In any case there may not be a Mercury Energy left to sue when all this is over: Although they are in theory worth just under a billion dollars, much of that is buried in the ground (and some of it just got reduced in value from "power cable" to "scrap metal") and up on poles, so trying to keep the central city alive for another two months may finish them financially. Just before the first of an avalanche of lawsuits against them began, Mercury made a last-minute bid to stave off problems by giving people direct access to Mercury to lodge compensation claims, at the moment it's uncertain whether this will have much effect. According to a story in the New Zealand Herald (Auckland's largest paper), the power problems go back to the old state-run Auckland Electric Power Board (AEPB), the immediate predecessor of Mercury Energy. Former AEPB engineers told the Herald that the cables needed to be replaced in the early to mid 1980's (in the early 1970's the cables were expected to last 10, perhaps 15 years if people were lucky). Mercury Energy has full records of meetings, reports, and discussion papers in which this problem was addressed, but has declined to make them available even though they were once public records, probably out of concern that they'd make a prime smoking gun for use in various lawsuits. According to the Herald, the issue of replacing the increasingly dodgy cables came up again and again, but was ignored by the board until Mercury inherited the problem in 1993. When the old power boards got restructured, a profit-driven mentality took hold with the idea being to drive the plant as hard as possible while performing the minimum of routine maintenance. I'm not sure how much of this is just 20/20 hindsight, everyone asked about the matter seems to have known that the cables were past the end of their effective life, but noone did anything about it. A Mercury board of directors member has claimed in a TV interview that they had no idea that the cables were dodgy: "there was no alarm sounded whatsoever". I'm putting my money fairly firmly on "We tried to tell them, but noone listened". Looking a bit further afield than the Auckland central business district, all of Northland is hanging off cables run by another power monopoly, Transpower, which are even dodgier than the ones which failed in Auckland. An engineer who helped install the power network in the 1950's described the overhead cables feeding the north as "totally shot" and "in need of urgent replacement". The closing of the white elephant Marsden Point power station in Whangarei several years ago means that when these cables fail, there's no way to provide power to all of Northland (the Marsden Point station consists of Marsden A, officially mothballed (meaning partially dismantled and left to rot), and Marsden B, which was never finished, everything just got left the way it was on the day work was stopped). Although there are new power stations such as the 400 MW Otahuhu power station (containing the worlds largest single shaft gas/steam turbine and generator unit) being commissioned, there's nothing north of the Auckland choke point. Hopefully the government enquiry will extend to looking at future problems in supplying power further north as well. My concern over this and the fact that I'm writing this from a location north of Auckland which is supplied by the aforementioned totally shot power cables is purely coincidental. This kind of outage brings home the fact that electricity isn't a simple commodity like clothing or electrical goods where an interruption of the supply doesn't have any significant consequences. Electricity is an essential requirement for modern life. If something goes wrong, it isn't just an annoyance for the company shareholders, it's something which affects everyone. This means that companies supplying essential services like electricity (and water and other similar services) can't be run like standard companies where the consequences of poor decision-making are restricted to the end-of-year profit and share price, but need to have substantial extra capacity and redundancy to provide a good safety margin. The current economic wisdom appears to be that privatisation is a Good Thing and everything else leads to waste and inefficiency. This was borne out by Mercury: They cut maintenance, cut the workforce, deferred investment in new plant, made record profits, and then knocked out the largest city in the country. Even if there's a requirement to make a profit, a discussion with a specialist in risk management indicates that there may be ways to ameliorate the cost of ongoing maintenance and upgrading. Consider the city of Wellington, which is run as one of the larger corporations in New Zealand. Wellington has an extensive and complex risk management strategy in place which is used to reduce insurance premiums (insurance companies can see the risk management in action and therefore reduce their premiums). Apparently this allows the city to run a good maintenance program without incurring excessive costs. In contrast the risk management strategy adopted by AEPB directors appears to have been early retirement. The Failed Cables ----------------- Two of the cables in question have copper conductors inside a pressurized nitrogen jacket and are referred to as impregnated pressure (IP) cables (apparently we're one of the few countries which still use these, they have a reputation for being rather unreliable and became obsolete soon after they were installed). Installation work on them commenced in 1947, so they're over 50 years old and although they're supposed to have a life expectancy of up to 60 years, in real terms they were past it 15-20 years ago (more on this below). The other two are oil-filled and are referred to as self-contained oil-filled (SCOF) cables. Installation commenced in 1975 (the installation, commissioning, and testing can stretch over several years, so the dates quoted can vary by a few years). The oil-filled cables are supposedly rated at 60MW each, the gas-filled ones at 50MW (although in practice they don't have anywhere near this capacity, more on this below). The central business district consumes about 150MW of power. The intial suspicion was that the El Nino summer has dried out and heated the ground so that vibration and ground movement (shrinkage) have damaged the cables. With daytime temperatures as high as 30C (86F) and humidity up to 75%, air conditioning loads have soared (at one point it wasn't possible to obtain fans in city stores). Ground temperatures have also been very high, accelerating the ageing of the cables. It was also susoected that variations in load were causing dynamic magnetic field changes, so that the electromagnetic forces between the three conductors and the steel pipe jacket (more on this below) caused the core to move around inside the pipe (this may have been the reason why the later cable failures occured, they would have been severely overloaded when the first cables went). Also, due to very slight temperature gradients around the cable, the differential thermal expansion would cause various bending and warping forces, however the exact cause(s) of the failures won't be known for certain until the damaged cables have been subjected to lab analysis. As it turned out, this was more or less what happened to the various cables (see the enquiry results further down). The oil-filled cables consist of three central copper/aluminium conductors inside a fairly sizeable pipe filled with an insulating filler, with three ducts carrying the pressurized oil dielectric consisting of tens to hundreds of thousands of litres of oil contained in a typical pipe. The much older gas-filled cables are generally similar but have a much lower capacity than the old-filled ones, and substitute nitrogen pressurized to 200psi for oil. In both cases the gas/oil isn't used so much to cool the cables but to improve its insulating properties. The copper/aluminium conductorss are each about 8cm in diameter core, and are wrapped in 200 layers of special oil-impregnated paper tape, followed by another copper sheath, and then a spiral wrap of 5mm bronze wire. These three wrapped conductors are then twisted together and inserted into an outer sheath filled with insulating filler with ducts for the oil or gas. The outer sheath consists of layers of lead, plastic, aluminium, and then more plastic. At each cable join is a splice which has temperature sensors, oil flow rate monitors, and pressure monitors to check oil conditions. Once the initial cable is installed, it's evacuated to both leak test it and remove any contaminants (for example when there's no gas present the oil from the paper tape which can leak into the gas duct in the gas-filled cables). Then the ducts are flushed with dry nitrogen, evacuated, and finally pumped full of high tension oil or gas under vaccuum to force any remaining contaminants out. For the oil-filled cables, the oil is first sprayed into a holding chamber at high temperature from hundreds of nozzles to get the maximum surface area, the high temperature causes all the crap to boil out so the good stuff which is left can be pumped into the pipeline. Once the ducts appear to be free of contaminants, a 5000V tester is used to ensure that none of the conductors were crossed when the cable was re-joined. After this, a high-voltage test is carried by solwly ramping the voltage up to at least 30% above the normal voltage. This is left for awhile, and the cable is then slowly powered down. This test is carried out for all conductors, after which the cable is hooked up at 110kV but left with no power flowing for at least 12 hours. Finally, the load is gradually increased over the next 12 hours, with the three cores typically carrying 600-800 amps of three-phase power. Both ends of an oil-filled cable typically have large holding tanks of oil, with pumps which are reversed every 6 hours so the oil oscillates back and forth in the cable, eliminating hot spots and spreading the heat over a large area. In the last hour of the cycle, things are slowly run down to get zero oil velocity, then everything is reversed and slowly run up again to move the oil in the opposite direction. Performing the repairs is difficult because at each point they have to sink a shaft down to the cable (digging out cable up to 10 metres from the fault point), line the shaft it to avoid the walls caving in, pump out any water which accumulates, and then repair the cable in the rather cramped conditions. Closing up the pipe after repairs is a special task in itself because as the pipe is filled with oil and paper it has to be done with special equipment and takes 8 hours to weld one section of pipe. If the pipe is too hot to touch 10cm from the welding, they have to stop and let it cool before they can continue. After that, they have to go through the lengthy refilling process described above. Within 10 years of the gas-filled cables being installed, they were experiencing 1-2 gas leaks a year, with each failure taking several weeks to repair. Luckily both cables never failed at the same time. In 1995 (with the oil-filled cables in place), both gas-filled cables failed, but the oil-filled ones held up to the load, which was rather less than it was by 1998. According to a former AEPB engineer, the problems with the oil-filled cables was caused by a combination of the cables being buried too deep and close togather, and high thermal resistivity of the material surrounding the cables due to the dry, hot ground. Because of this, the cables should have been derated to around 50% of their actual capacity at the time of the failure. The cable manufacturers also stated that they were being run well outside their design limits (more details on this below). Fixing the Problem ------------------ Mercury have a $120M, 9.2km tunnel under construction which (had it been finished) would have brought in the required power, however it passes under the buildings which are now affected and during the planning stages the owners of the buildings raised all sorts of objections to it (the project was initiated in 1994, but construction didn't start until May 1997). When completed, the tunnel, which runs between 20 and 100 metres underground, and carry power and communications cables into the central city. The delay in starting the construction work, and the fact that Mercury delayed things while they got other organisations to help fund the project by selling them space in the tunnel, held things up for nearly two years, pushing back the 1997 completion date by at least another 1 1/2 years. I see a long period of finger-pointing to follow. As a relief measure, Mercury ran 9km of double circuit overhead cable along concrete pylons hastily erected along the rail access corridor into the central city. Noone dared to raise any planning objections this time. The work was originally expected to take 6 weeks to complete using a mongolian hordes approach with materials garnered from all over the country and cost $5M, although like all of Mercury's estimates to date the completion time ended up being stretched out by another few weeks. This timetable was also useful as a real indication of how repairable the oil-filled cables are: Mercury gave the time to restore power to the central city as the time to install the stopgap cable, indicating that although they wanted to try repairing the oil-filled cables yet again, even they don't believe that it was possible any more. Mercury have also indicated that the stopgap cable would be expected to hold for 18 months or until the tunnel is ready, which indicates that they're not planning to get anything else done until the hoped-for completion date of the tunnel. Amazingly enough, the emergency link was completed in less time than expected, with a warren of feeder cables at both ends being run to and from various substations in the vicinity. The rail tunnels were hastily fitted with what one observer described as heffalump traps, and Mercury announced over the radio that people could turn off their generators. The power stayed on for nearly two hours after the all-clear announcement was made. By the following Monday, everything was more or less back to normal, with power being supplied by the emergency feed. The Aftermath ------------- The government inquiry into the power supply failure was completed relatively quickly, and was published on 8 May. The inquiry was carried out by Integral Energy, a large Australian power company. The report found that Mercury's planners made judgements on cable loading based on information provided by the manufacturers rather than the actual conditions at hand (that is, they ran the cables at their theoretical rating rather than the actual rating under the conditions they operated in). Although the cables supplied met the requirements set by the (former) AEPB, they were operated in conditions well outside those given in the requirements. The gas-filled cables were installed "contrary to good engineering practice", with bad ground stability in one location leading to a continuing series of faults at that location. In addition since Mercury's planners regarded the gas-filled cables as inherently unreliable, they didn't place much emphasis on maintaining the cables and didn't investigate the cause of the repeated failures very closely. At the time of the investigation, Mercury were not utilising various alarm functions connected to the gas-filled cables. As has been mentioned earlier, Mercury considerably over-rated the cables for the conditions they were being operated in. Although the oil-filled cables have a nominal rating of 60MW, ground conditions meant they had a much lower actual rating, so that the cables started to overheat when loaded to anything more than half this rating (this indicates that even with all four cables fully functional and supplying a normal load to the central city, the cables were already overloaded and overheating). This overheating couldn't have been detected as part of Mercury's maintenance program. According to the report, Mercury didn't comply with the manufacturers recommendations for routine testing and maintenance. As a result once the gas-filled cables failed the first oil-filled cable failed due to thermo-mechanical reasons (the higher-than-allowable operating temperature allowed the conductors to move, causing a joint to be compressed which resulted in it failing), and the second oil-filled cable, now carrying the full load, simply died from thermal runaway (or to use trade union parlance, it went out in sympathy). The enquiry completely avoided looking at Mercury's peculiar management structure, covering only the technical aspects of the failure. It's unlikely whether anyone will ever know the total cost of the outage, but it's estimated that it cost Mercury around $150M, which is two years worth of after-tax profit (their yearly operating revenue is $600M). Bits and Pieces --------------- Here's a panoramic photo of the central city by night: +-------------------+ | | | | | | | | +-------------------+ In the foreground you can't see the town hall, with the Aotea Center and council building in the background. To the right you can't make out the side of the Sky Tower, with the casino beneath it. Here's another shot, taken from down by the waterfront: +-------------------+ | | | | | | | | +-------------------+ In this one you can't see Queen Street and the businesses up either side, with the Coopers and Lybrand tower not visible to the right. I'll put more shots online later if I can find somewhere to power a scanner. The power outages did bring out some good things. After the power had been out for about half an hour on the first day, someone mentioned that the fridges downstairs wouldn't be powered. In the spirit of true cooperation and self-sacrifice, everyone immediately rushed downstairs and saved all the beer from getting warm (for UK readers who may not understand the significance of this: Cold beer good, warm beer bad). Auckland joke (you probably have to be a NZ'er to get this): Q: If there are power shortages, which will you keep running, the cappucino machine or the air conditioner? A: Both. More jokes: Q: How many Aucklanders does it take to change a lightbulb? A: Does it matter? Pickup line for Aucklanders: "Would you like to come up to my room and see my work?" Q: What did Aucklanders use before candles? A: Electricity. Because of the Y2K problem, various people are planning on taking vacations away from electronic devices in December 1999. Central Auckland is looking like a good place to take this vacation. Various Odes to Mercury Energy ------------------------------ This isn't as good as I was hoping for, but here it is anyway, to the tune of "Day-o" (I think that's what it's called): Powwwwwwww eeeeeerrrrrrrrr! Pow-ow-ow-er! Workday come an' me Wan' me power. Hey mister Mercury man, Gimme back me power Workday come an' me Wan' me power. Six-week, seven-week, Eight-week outage! Workday come an' me Wan' me power. Powwwwwwww eeeeeerrrrrrrrr! Pow-ow-ow-er! Workday come an' me Wan' me power. I work all day to build up me business Workday come an' me Wan' me power. Now Mercury put me out of de business Workday come an' me Wan' me power. Powwwwwwww eeeeeerrrrrrrrr! Pow-ow-ow-er! Workday come an' me Wan' me power. This one, contributed by Nutta McBastard , is much better: BLUN-DER-ING MERCURY (TO TUNE OF QUEEN'S 'BOHEMIAN RHAPSODY') Pow-er, just went off again. Flicked the switch right by my head. Lights were shining now they're dead. Pow-er, it had just been on But now it's going to black out right away. Pow-er, ooh-ooh, Not enough to run the lights, Don't have power now, won't have no power tomorrow, Powers off, powers off, 'cause cables died in-toto. Too late, my dinner's cold, Cold spaghetti, warm white wine, Oven's powerless all the time. Goodbye frozen produce, you've got to go Fridge defrosting means you rot, no time at all. Pow-er, ooh ooh, Mercury you make me cry, I really wish you'd never been here at all... (Guitar solo) I see a little UPS in place of pride, Battery power, battery power! Pass the fuel for the generators. Diesel does just nicely But it's very very dicey, BANG Generators? Generators? (GENERATORS?) Generators they don't go, (Burning generators just don't go. ) I've got no customers, somebody help me, He's got no customers, get him cash immediately. Economy's gone to crap And this city's on its knees. Power comes, Power goes, Nothing ever works. Have power? No! It's blacked right out again. (It not go!) Mer-cu-ry! (They cannot make it go!) Mer-cu-ry! (They cannot make it go!) (They cannot make it go!) (They cannot make it go!) (Go? No no no no no oh no no no!) Mer-cury-ia, Mer-cury-ia, Mer-cury-ia make it go! Mer-cu-ry has devils for Directors. Me... Poor me... Poor me! (Guitar solo) Directors blunder and believe it's alright? They want to take over every Power Co in sight. No way-mate! Let's hang the lot - YAY! The chair would be better, But it just won't work around here. (Guitar solo) Power really matters, Consumers they can see. Power really matters, But not to Mer-cuuu-rryyy. Acknowledgements ---------------- This writeup originally started as a page of notes covering an afternoon power cut. By the time it had grown into the current lengthy saga, I'd lost track of who had contributed what, and when (and even the dates were a bit hazy, since it was only expected to last a week I used references like "Wednesday" to specify a time). Suffice to say that lots of people have helped in bringing you this information.