What's Behind a Banking Glitch, Anyway?
What caused the epic banking glitch that made some billionaires for a day?
(SPOT.ph) The recent BPI system failure resulted not just in collective frustration and many millions of man-hours of lost productivity, but affirmed a continued distrust of the electronic systems that drive modern banking. So what went wrong? We interviewed computer systems specialist Fernando Contreras, Jr. to understand the technology that underpins every transaction you make, regardless of which bank you entrust your money to.
Contreras speculates that it was a botched database sync that caused the glitch that made instant billionaires of some while apparently draining the money pots of others, and eventually resulted in the entire system being taken offline. The error in syncing jumbled up the records of who had what amount of money, and had done which transactions. The delay in getting the system back online suggested that it was likely the core database had been corrupted and had to be rebuilt.
It’s hard to imagine that our hard-earned pesos are just numbers in a big catalogue, but this is how every bank in the modern world works: Imagine a digital version of an index card with your name and your credit amount written on it, and every transaction you’ve ever made recorded and stored in a big electronic filing system. In the Philippines, these are known as the CASA (Current Account/Savings Account) databases. After banking hours, during the wee hours of the morning, these are backed up, the same way you back up your important files (you do, don’t you?). An example of a database that most people are familiar with is your iTunes Library—and many are familiar with how a big library used to crash and needed to be rebuilt. In this case it was probably a wrong batch program: For example, someone tried to run a script manually and made a mistake.
Most banks still use “mainframe” computers, the kind that take up an entire room, and are accessed by a terminal with a proprietary connection. The reason they do this is to protect from hackers; these giant computers are not connected to the Internet. By omitting the TCP/IP protocols that are built into almost everything these days, from your router to security cameras to some refrigerators, a hacker in Russia or North Korea cannot get into the system because they are literally not plugged in. The problem, however, is that they are clunky, slow, and can become easily overloaded. Examples of this include Systematics, VSAM, and the like.
Newer systems of relational databases (called RDBMS, for Relational Database Management Systems) have the advantage of speed, flexibility, and can scale better as new customers are added. However, they are connected to the Internet, so this means setting up firewalls and constantly monitoring them. Some will remember that recently Amazon’s S3 storage system crashed, and sent large swathes of the Internet dark—ultimately it was traced to a typo on the part of one person trying to debug the system that caused a cascading error that took multiple servers down.
Despite this there’s a strong push now to move toward using modern databases such as Oracle, Microsoft SQL, IBM’s Informix, and similar. You may not have noticed it, but ATM downtime has actually been decreasing of late, and heavy penalties are levied on partner banks for downtime. The most common problems for ATMs, as with credit card machines and so on, are “last mile” problems: basically, the telecommunications infrastructure.
But online banking, by definition, requires the bank systems to be online, and to use the newer systems, which banks are reluctant to do because of increased need for security, and a greater chance for glitches like the one we just experienced. The old mainframes also don’t do “horizontal scaling” well, which means being able to increase capacity without having to add to or improve the hardware. They also also don’t do real-time transactions well. Banks tend to be very conservative and stick to an older, stable system that they know well. Unless it’s a deliberate hack, most banking system failures are caused by human error or overload during peak usage.
While it’s true that more Filipinos should save in banks rather than under their mattresses, and many of them aren’t computer literate and can’t access online banking, there’s actually a good deal of trust in abstract stored value: Globe and Smart both offer successful electronic currency in the form of GCash and SmartMoney. Companies like Ayana, Voyager, mBTC, and others are trying to push cashless transactions. The driver here will really be infrastructure—while we have connectivity in a significant area of the country the costs are still too prohibitive for a sari-sari store to accept cashless transactions.
The next frontier for the banking world, aside from better databases that are still secure but fast and scalable, are blockchains. Bitcoin is probably the most famous, but there are others operating along the same principles. With Bitcoin ledgers there are no modifications or deletions—everything happens in real time and there is transparency and resiliency. But this technology is in its infancy and has the possibility of bypassing traditional banks altogether.