Article here from DataMonitor on virtualization talks to the more general situation where businesses need to understand the benefits of  technologies versus the overhead they incur. In my experience, many IT groups simply do not factor application and hardware asset control & management into their solutions and environments. More often than not systems monitoring and controls are applied (via duct tape, etc!!) after deployment which in turn leads to an inconsistent and fragmented control environment, and hence trouble with internal compliance and regulators alike. The challenge with virtualization (and compute/data grid technologies) is that they need to be considered at both a business level AND an enterprise level, if the full benefiits are to be recognized (and controlled).

Over the past few months I have been asked a lot about what technologies can be used and applied to resolve internal compliance & operational risk needs as well as regulatory requirements.

The common problem many banks face is that they are trying to focus on each requirement in isloation and their approach is often one of wallpapering over the cracks together with a healthy amount of duct tape. Some banks do have decent application, middleware and hardware monitoring tools to identify performance issues and outages in real-time. This often results in “how fast can I switch/failover to my backup machine ?”

My point here is that these regulatory and compliance issues should not be addressed retroactively. THERE IS A REAL BUSINESS OPPORTUNITY to provide a highly performant, competitive and controlled business/IT environment that FIRST makes the banks more competitive and successful in the marketplace, WHILE addressing the compliance, operational risk and regulatory requirements.

When you look at the set of requirements from across the various regulations such as Sarbanes-Oxley (SarBox), RegNMS, Basel II and MiFID in particular, there are a number of really tough issues to crack, but they really should be approached by looking at the business criteria for success:

• Market Making Availability (robustness)- regulators demand that market makers need to be “always on” by ensuring that they are continuously connected to the market and provide low latency responses to pricing and execution transactions. A good example is in equity derivatives where the trading desk needs to provide liquidity into one or more exchanges (ie be connected and responsive to RFQs, etc).  More often than not, businesses use traditional “monitor and control” mechanisms to identify in pseudo real-time performance issues and/or outages, with an associated “failover” process. For this, compute and data grids can be used to provide a robust and continuously available platform which runs independent on the state of any given physical asset in the infrastructure. So, if one node of the grid goes down, the remaining processes in the grid will ensure transactional and information integrity is retained. Additionally, if additional throughput is required, many grid solutions will allow for linear scalability as more CPUs and processes are brough-online automatically (or manually). Why would a business not want that ?!
• Competitiveness (pricing and performance)- joined closely to the hip of availability, being competitive in the market place is an important issue for regulators where market participants provide the appropriate levels of liquidity at prices that the market can tolerate. But this is most certainly a primary business issue and where banks can differentiate themselves in the marketplace from their competition. Part of being competitive in the online marketplace requires that the front office applications in particular deal with large volumes of tick data, order flow and trade executions. Providing these services to the market at low levels of latency is particularly challenging as there are often many functional steps to creating and publishing prices and executing orders in the marketplace, especially with derivatives. Using the example of algorithmic trading of equity derivatives, where the business relies on being able to create and deploy (and remove!) trading strategies “on-the-fly”, the infrastructure needs to provide a robust high performance environment that can be flexible to changes in functionality (without having to rely in off-hours to make functional changes). One of the nice things about certain technologies such as Javaspaces (or commercially, Gigaspaces) or Tangosol is that they can be configured to hold (and replicate) objects in cache to minimize messaging latency issues around serialization-deserialization between remote application processes. Additionally, the compute time of complex pricing routines/libraries can be optimized when appropriately run across a compute grid, Monte Carlo simulations being the classic example. In both examples, there are a number of ways in which all the data and transactions can be persisted in real-time, or by write-behind. They key to all of this though is a well thought out architecture, use of efficient application design patterns and framework, and a boatload of performance tuning.
• Auditability & Repeatability (control)- in order to satisfy both internal compliance/audit controls as well as regulatory requirements, systems need to be able to provide enough of the relevant data and information that went into any given transaction AND be able to repeat the calculation or process that created the price. For example, when pricing a client portfolio, not only does the application need to have a record of ALL of the data that went into the calculation (trade attributes, market data, sources, spreads, etc), it ALSO needs to be able to reproduce any calculated results - this means that the analytics and other code- and configuration- dependent processes need to be captured and kept for future re-calculations. Therefore, retaining actual binaries, libraries and configurations at the time of execution as well as the actual data itself is key and is very often overlooked when it comes to providing true repeatability. In order to implement such a capability, very specific design patterns and application configurations need to be considered during design & development. Additionally, SarBox requires controls around roles and permissions to be defined and enforced. Although these types of controls seems to go hand-in-hand with auditability & repeatability, what do you actually really do around things like Excel spreadsheets which proliferate trading, risk, and operations groups ?
• Price Transparency, Best Execution and Reporting (data)- generally speaking, some of the latest requirements (especially RegNMS & MFiD) demand that banks capture and warehouse a broader and deeper set of the data that went into prices and transactions. This in turn means that banks need to be creating broader and more sophisticated data warehouses that can model and persist cross-asset and market information, and that support multi-dimensional analytics and reporting (e.g. OLAP). Generating this additional volume of information throughout the trading day will place massive demands on application development groups, the supporting infrastructure, let alone the poor persistence guys !  Related to price transparency and best execution, capturing all of this data in a liquid market would most likely meet the needs of regulators, but what about illiquid markets where there might be a void of market information ? Most likely, the repeatability of the process (as defined above) is all you can do. Large in-memory storage technologies and cache replication certainly help in dealing with vast quantities of data, but they ultimately need to be persisted in data warehouses, with ever increasingly diverse reporting applications being built to access, slice and dice the information. These are tough (and expensive) issues to solve across an entire enterprise, so no wonder many financial institutions are pushing back. That said, there is some business value in having broader data sets being captured across all trading business: richer market datasets can improve the quality of market risk models that use historical market data and derived information such as trends, volatility, correlations, etc. Both trading research desks and enterprise risk management functions could use this. The question is what is the cost-benefit ?

Solving these issues is no trivial task, but the various technologies and techniques in the “Large Scale Computing” (aka grid) domain, can help out vastly. However, all of the hype around the potential benefits of “grid computing” (which can be enormous) has to be moderated against the architecture, controls and configuration required to manage these additional assets — precisely the issues the regulators are asking to be solved. What business-driven IT groups should be thinking about should really be business driven requirements around providing reliable, competitive and controlled services.

Decent article here from Dr. Dobb’s Portal on how to scale agile projects.

In our experience, on a discrete project basis, a small team of senior engineers will generally always outperform larger “less senior” teams (depending on the success criteria of course!) especially when working with trading, pricing and risk management systems and their ever-changing needs. The real problem in scaling becomes evident when different groups need to integrate their part of an overall solution. In too many cases in Financial Services, each component of a Service Orientated Architecture for example, is so generic (ie not a fully encapsulated service) that is not actually that useful on its own. And end-to-end programs/workstreams that try to stitch these components together are incredibly guilty of taking the ‘command and control’ approach to planning, design and implementation. This forces any agile work down into the grass roots, and limits its adoption across an organization. It can also seriously hinders client-developer communication and feedback. Release planning and integration also become a nightmare and highly frustrating for the agile team (I assume non-agile teams are also frustrated but that is modus operandi).

I am glad to see Collaboration getting a good profile in the article, as it is absolutely paramount to successful agile development, let alone scaling it. We have had a lot of success both internally and with clients in using a wiki to provide complete transparency, continuous updates and real-time issue/risk management — but it is only one part of the solution. Executing larger scale projects needs a high degree of interaction and collaboration on a number of levels including client-driven planning, continuous communication & feedback, daily stand-ups (team and Meta flavours), and in-your-face planning & retrospectives. For distributed teams, even if the teams have actually successfuly worked together before, the initial planning (iteration zero?) phase MUST be physically done in person (a la Agile Modeling Driven Development, if need be) – to establish a real working relationship and sense of trust and understanding, especially for those darn conference calls you need to work your way through ….

The New York Times ran an article the other day on how the number of patents FS companies have been filing has been growing at a crazy rate.  Since 1997, the number of patent applications has grown from 927 to 6226 in 2005, and the rate continues to increase. It seems that many of these are software-based around trading, pricing and risk management systems, in addition to product-based patents for things like credit cards, ETFs and exotic derivatives.

The article goes on to describe how specifically technology innovation is not only a competitive tool, but its patenting will/could lead to some major legal battles down the road.

Matt – what ever happened to your joint client / UI framework patent(s) ?!!

As I mentioned before, our task-driven estimation on our project iterations drive burndown charts that we continuously update and expose through our wiki.

I found this article on Big Visible Charts which has some great ideas for test and story-driven development… very intuitive. One of the charts includes a data series that shows changes in stories, and the resulting change in expected ship date of the product.  Related to this (kind of), we have really caught the attention of our client stakeholders on one project by adding a “descoped” effort onto our Burndown as this really tells a story of what going on with the delivery team (in this case, getting bogged down with wrapping legacy code and integrating with external systems). Clients generally do not like things being descoped as they are normally the ones that try to add to scope and effort !! Still, the “big visible chart” started a conversation …. which is the point.

Looks like the Bank Of New York (BONY) has been busy this year re-inventing itself, trying to take advantage of  “..the growth of the hedge fund industry, the rise of high-speed electronic program and automated trading, and the increasingly large role of private equity in the overall economy.”

Their transformation (so far) includes:

• Sells its Institutional Agency Brokerage Business to BNY Convergyx — a BONY spin-off that includes GTCR Golder Rauner, a private equity firm, and Eze Castle Software, a trade order management systems provider with a large hedge fund clientele. This results in BONY:
  • Keeping their FX/securities lending & payment, treasury management, asset management and private banking businesses
  • Gaining hedge fund flow from the Eze Castle Software entity
  • Expanding Hedge Fund Administration group
  • But, still does not have Prime Brokerage arm.
• Sells retail-banking business to JPMorganChase (300 branches worth $3.1bn)
  • Bank of New York’s remaining processing business (could be worth $26 billion)
  • And then announces that it will acquire JPMorganChase’s corporate trust business (for 2.8bn), as part of the deal.
  • [When I was at JPMorgan in the early-mid '90s I seem to remember pre-Chase JPM selling its Securities Trust & Information Systems (STIS) to BONY, so they have made some good business off the back of JPM/C's cast-offs!!]

So, it looks like it is focusing on its set of securities servicing, asset management and private banking businesses. I wonder what the “Office Of Innovation, headed by CIO Kurt Woetzel, will do to further “drive product innovation across Global Markets” …I would also love to know what type of grid they run all this securities processing on ….

A TechRepublic Blog has a decent summary of the various types of virtualization methods that can be deployed: hardware-virtualization, para-virtualization and OS-virtualization. The blog is based on this paper

With over $5.5billion in earnings in the 2nd quarter, Bank Of America is the most profitable financial institution in the world .. articles here:  Economist and Bloomberg

Now BofA is only $4billion behind Citigroup in market capitalization (BofA is now $234billion), versus being behind by $40billion a year ago. Interesting to see that BofA makes most of its money in the US through business and consumer banking as well as a large share of the loan-syndication market. Their online banking, credit cards and wide range of products  (and good old customer service) seems to be way ahead of the competition. They also have one of the largest Private Banks. Citi on the other hand, has a narrower retail business in the US (CA, TX, NT), sells more complex investment products, and has a bigger investment bank business, as well as global operations. Their acquisitions of with Fleet Boston Financial Group ($47bn) and MBNA ($32bn) helped to close the gap on Citi.

I wonder if BofA would expand its retail (and IB?) business globally, using a model like the Royal Bank Of Scotland (where it acquires local/regional banks, putting them onto its “IT platform” together with a carefully crafted re-branding exercise (e.g. Citizens bank’s logo is RBS). If the BofA retail IT infrastructure is that good, how scalable is it outside the US ? What is their appetite, if any, for non-US retail banking ?