Notice:
  The views and information expressed in this document reflect the opinions and experience of the author Robert C. Pelletier.  Neither CSI nor the author undertake or intend to provide tax advice or trading advice in any market or endorse any outside individual or firm.  All recommendations are provided for their informational value only.  Readers should consult competent financial advisors or outside counsel before making any software purchase or investment decision.  CSI does not stand behind or endorse the products of any outside firms.

Copyright (c) 1998 Commodity Systems Inc. (CSI).  All rights are reserved.

The Truth About Data Soaring Stock Market Calls for CSI Format Change

The Truth About Data

A truth:

    The more serious the analyst, the more careful the consideration of data. This is as true of white-coated scientists in laboratories as it is of investment professionals on Wall Street. Another truth: The more careful the consideration of data, the more troublesome anomalies will be found.
Over the years I've met many CSI customers who stake their reputations and their livelihoods on the market research they perform. I have come to know them because they had concerns about data which they felt only I, as president of the company, would fully appreciate. Whether or not others could have helped, I was glad for their calls, as they have helped me stay in touch with the needs of our users. It occurs to me that, although most CSI customers do not ponder the validity of every data point, all might benefit from the research we have done and the explanations we have formulated at the behest of our most ardent researchers.

Missing Data

    One of the most common concerns involves so-called "missing data." Null fields can, indeed, be a problem when algorithms require daily comparisons or accounting. If a study is intended to perform a simulation based upon all trading days, ALL trading days MUST be present. Anything less would be inaccurate. However, missing data is sometimes simply a part of the factual record of a commodity's trading life. CME Pork Bellies, for example, were not traded for the brief period between August 23, 1963 and September 20, 1963. Alberta Barley did not trade between February 28, 1989 and May 24, 1989. White Sugar, MidAmerica Treasury Notes, the French Franc, Unleaded Gasoline, the Mexican Peso, London Tin and COMEX Silver all have trading gaps that are reflected as missing days in data files.

    One poignant concern came from a man who was comparing databases and wanted to know why CSI was the only one of three data services with a gap in silver prices for three days from August 16th 1971 to August 18th 1971. We frantically researched our files to find that, low and behold, there was no trading in silver during that time. Back in 1971, our records show, President Nixon imposed an embargo on finished goods that affected metals and other commodities, forcing the exchanges to close for a few days until the impact of the President's decision could be absorbed by the emotional public. Where our competitors got their prices, we'll never know. We do know you won't find fictitious data points in the CSI database that were invented for the sole purpose of making the series appear credible. As far as we know, you won't find any inappropriate gaps in your CSI data.

Out-of-Range Settlements

    We have discussed this before, but continue to get many questions regarding closing prices that are above or below the day's high and low. Typically only users of Unfair Advantage® have queries about out-of-range closes. This is because QuickTrieve®, our original downloader, and many products for market analysis do not allow for closes outside the trading range. Unfair Advantage, however, can report out-of-range settlement prices, regardless of trading range violations, because they are a true part of the market data.
The confusion about out-of-range closes stems from the idea of having a simple "closing" price for commodity contracts. Although the "closing" price could represent the last trade, at some exchanges it could be a calculation decided by a committee after the close of trading. Low volume, far-out contracts are most likely to have adjusted settlement prices.
Unfair Advantage users can present the data in 1) actual exchange released form with highs, lows and settlements left untouched, 2) with compromised closes that ignore outlier settlements and force the close to lie within the high-low range, and 3) in compromised high and low form with accurate settlements and possibly a high or low that has been modified. Here again, when the analyst adheres to the true market record (by choosing actual), apparent anomalies are added to the data series. But, like missing days where no trading occurs, out-of-range settlements are a part of the record that analysts may benefit from accommodating in their evaluations.

Two Closes for LIFFE

    The London LIFFE exchange is particularly prone to confusing closing prices because of the way they blend computerized APT data (After Pit Trading) with floor trading data. The APT session begins about 20 minutes after the pit session is over and lasts about 90 minutes. The data supplied by the exchange is as follows: The open from the pit session, the high and low statistics compiled from all trades from both the pit and APT sessions, and the official settlement price, which reflects the close of the pit session.  The close from the APT session is also available, but it is not "official," and is therefore not quoted on our system. Neither our price quotation, nor any others that we know of, can accommodate two closing prices for the same contract. We hope to eventually add a field to Unfair Advantage to accommodate LIFFE's double closes.

Required References

    If you were to visit CSI's facilities, you might be surprised to see source material neatly cataloged and arranged on shelves that line the walls of the data department. The material is in the form of microfilm, exchange bulletins, newspapers, old out-of-business newspapers and bound volumes of trade reports. We have materials that were published as far back as 100 years ago. We can back up and verify just about any questionable report and certify relatively ancient data accuracy. We save just about every possible data record so that our customers can get the most for their money. The fact that we publish every error we commit is a testament to our devotion to doing the job right.
If you also use another source and a discrepancy is discovered, 90% of the time CSI will have the correct reading. None of our competitors are brave enough to publish their mistakes, and we would be happy to challenge each and every one of them to a contest on accuracy. We ran an add several years ago showing a side-by-side comparative record of errors committed by CSI and one of our competitors, but after a while S&C Magazine refused to print our reports.
While it is certainly true that our most serious customers provide us the greatest challenge in providing the highest quality of data and support, it is also true that careful consideration of the data is what sets CSI apart from all others in the industry. Anomalies will always be found in data, simply because we live in a world that is not perfectly structured. It is important to know that these anomalies reflect our somewhat chaotic world, not errors on the part of your data provider. The truth is, we are privileged to serve such a dedicated and thoughtful group of investors and analysts.

Soaring Stock Market Calls for CSI Format Change

    Barring a major downtrend in tech stocks, the NASDAQ Composite Index will soon eclipse the CSI format's upper price limit of 262143. That is, if it hasn't already done so by the time this newsletter is printed. QuickTrieve users and those who use QuickTrieve-compatible software will note distorted readings for this index on all days the price exceeds 2621.43. CSI's Unfair Advantage software will not have such problems in its charts because it can accommodate number sizes beyond 8 million.
As this letter goes to press, new format specifications are on our web site for software developers to download and implement. These will extend the CSI (QuickTrieve) format open/high/low/close price range from 256K to one million. By the time you receive this newsletter, revised QuickTrieve software files will also be on the web for users to download and install.  We strongly recommend that everyone upgrade to the revised format once it is released. The new format will require changes to your QuickTrieve software and to any compatible software you use to analyze CSI-format data.
The CSI format has been updated a few times over the years and has changed from handling two-byte quantities (16 bits) to the new convention, which handles 20-bit quantities (2.5 bytes). Periodic revisions are necessary in any fixed-field format (such as the CSI format) to accommodate an ever-expanding range of positive number sizes. Developers who stay current with the CSI format will be able to process data with values of 999999 or less and should have no trouble reporting data from all exchanges.
Unfortunately, some software developers have been slow to respond to the expanded precision that CSI offers. Those who haven't upgraded their software at any time may be limited to expressing and processing data with a maximum quantity of 65535. If you are not experiencing problems, then your software may well be compliant. On the other hand, if you have difficulty charting your data with software other than UA, then your program is probably operating on the old CSI format. Hopefully, the developer of that program has already made an upgrade or is preparing one for release.
 Let's assume the NASDAQ Composite Index has already broken through 2621.43.  If you haven't upgraded your QuickTrieve software and/or your analysis software is subject to the original 16-bit number size limitation or the 18-bit version of the CSI format, please consider these options:
QuickTrieve users can track the NASDAQ Composite Index with CSI stock number 5740 (COMPZ). The price values in this series are quoted to one (rounded) decimal place, instead of two decimal places. This stop-gap measure will produce charts and data which are less precise than the standard NASDAQ Composite Index data (5799 COMP), but they will not be distorted.
Most Unfair Advantage users who store data in the CSI (QuickTrieve) format will note no distortion because of changes we transmitted to UA's factsheet and database control files. Those UA users who have protected the factsheet file from remote revisions should see Tech Talk for information on revising this file.
The exchanges are neither unaware nor insensitive to the effects of rapidly rising values on traders. The higher prices go, the less meaningful incremental changes become. We anticipate that the NASDAQ will soon change the minimum tick for its index and divide the values by ten, just as we suggest doing through Unfair Advantage. In either case, the change will be virtually transparent to the user.

    QuickTrieve users and compatible software vendors are asked to check the Patches page of our web site at www.csidata.com for emerging information and upgrades. These are provided at no charge. If you do not have access to the Internet, please call our technical support staff to request a diskette of the revised QuickTrieve software. A $10 handling fee will apply.