[Click here to return to journal index]
© Copyright 2003-2004 by Commodity Systems Inc. (CSI). All rights are reserved.
Website Links: This CSI website and Unfair Advantage system contain hyperlinks and automatic links to websites operated by parties other than CSI. No association with or endorsement of third-party websites should be inferred through these links. They are provided for your convenience and reference only. CSI is not responsible for and has no control over the content of third-party websites.
|
March 2004
CSI Technical Journal Volume XXII, Number 3 |
Page 1 |
Topics discussed in this month's journal:
Accelerating Data Distribution with Unfair Advantage® Tech TalkAccelerating Data Distribution with Unfair Advantage®
As president of CSI, I am fortunate to receive regular feedback from customers about our services, most of which is quite favorable, and all of which is appreciated. Unfair Advantage users sometimes tell me that the time required to distribute and correct daily data is somewhat longer than ideal. Acknowledging that in the data retrieval business, more can sometimes be less, this month’s Journal offers suggestions for speeding the process.
Unfair Advantage’s "Preferences" include a few settings that may be helpful in improving distribution speed and reducing delays. The choices represent a tradeoff between distribution performance and protection against corruption of the database that can result from hardware failure and software errors.
The tradeoff is necessary because, like it or not, computers experience problems from time to time. It’s unavoidable: The scanner you just installed causes your computer to freeze because it requires an updated driver; your new Windows XP might freeze unexpectedly until you replace that video card the installer warned you about; lightning strikes. The list of possible glitches goes on and on, affecting some computers more than others. UA lets you make certain choices that directly affect the amount of time that might be lost recovering from such problems and the time it takes to distribute daily updates.
Try the following software changes to get UA humming.
UA Speed-Up Tip #1
From the main menu that runs across the top of your Unfair Advantage screen, click “Preferences.” UA will display a list of categories of user settings in the left panel. From there, click the "Advanced 2" item. A slider bar displays on the right allowing you to select the degree of database corruption protection employed during data distribution.
When UA updates your database, the actual data on the hard drive is not immediately changed. Windows saves the work of writing to the hard drive in a prioritized task list. Depending upon the priority, Windows may stop everything until the task is done or it may wait until it is convenient or until there is nothing else that needs to be done.
The first choice on the slider, “Do not flush writes during updating,” makes full use of Windows File IO optimization to provide you with the fastest possible distribution. It tells Windows to give the actual writes of the database file lowest priority. This provides an additional advantage when updating multiple days, as UA can efficiently update the in-memory copy with all the days before Windows makes a single pass to write all of the changes to disk when everything is done.
So what’s the tradeoff? The downside of waiting to write files to your drive becomes painfully clear when your computer fails - even for an instant. The list of reasons customers have reported for sudden system freezes is dizzying. Who knew that plugging your digital camera’s memory card into your snazzy new memory card reader could cause Windows to freeze? We do now. Every time such a failure occurs before all data is written to disk, you’ll lose most, if not all, of your UA database. This is because those wonderfully fast in-memory updates aren’t written to the hard drive, and the jagged end of the file may contain random data that can literally destroy your UA database. Ouch!
The second fastest setting, “Flush writes as files are complete and employ a log file,” is a safer alternative. When UA has finished writing to a database file, it tells Windows to write the changes to the hard drive while it waits. UA runs slower, but all files are written with each update. Since the directory itself is kept in memory, opening and closing of files is pretty quick.
The tradeoff? This method is still subject to file corruption in the event of a power failure or hardware glitch during distribution. If the table saw you are using while UA distributes shorts out and throws the breaker in the middle of distribution, the directory with the UA databases probably won’t be up to date. It might show that the Corn database is 830,000 bytes instead of the correct size of 831,488. Those lost bytes are crucial and cannot be replaced except by getting a whole new database. Moreover, since the directory is updated as Windows sees fit, other recently updated files within the directory will also have an incorrect size. Depending upon your Windows version, the operating system may even delete these files or consider them “lost chains.” If this happens, you can use Unfair Advantage’s DB Integrity search to try to determine which files were damaged. This could be a couple of files or a few hundred, so you’ll have to decide for yourself if online replacement is practical.
The third setting, “Avoid lazy intermediate writes and flush,” tells Windows to use a higher priority for writing files to the UA database than it employs with ordinary text documents. This should result in the database files being written correctly and the directory list being more current in the event that new game you are installing while UA distributes freezes your computer. The directory list may be inaccurate, but thanks to the higher priority, it should only affect a couple of files. These files can be found and replaced using the DB Integrity Search facility in UA.
The fourth and safest setting, “Flush entire drive's file cache with each update,” forces Windows to write all changes to the hard drive immediately while UA is updating. Of course, that means that Windows has to constantly wait for the hard drive to do its work, increasing the time required for updating by as much as tenfold. Be aware that this will not only slow UA, but also everything else running on your computer at that time. You could still, theoretically, lose one database file if an interruption occurs after the database file has been revised, but before the directory has been updated.
The default selection is second from the top, “Flush writes as files are complete and employ a log file.” This setting is recommended for computers that are working reliably and for users who will not be too inconvenienced by an outage or two per year. However, for those users with rock-solid computer systems and a backup, the top setting is preferred. The speed improvement can be quite remarkable, but there is an increased likelihood of database corruption. On the other hand, if you like tinkering with your computer and are always trying new things, the third setting may suit you. If database failure is not an option for you, we can only recommend the fourth setting.
Some people have purchased hard drives with onboard memory caches. These certainly speed up UA distribution for all settings except the fastest, but you have to depend on your hard drive to write those 8 MB of updates to the hard drive when there is a problem. Using an IDE controller memory cache should result in fewer files being damaged for the same performance, provided UA is configured with a safer setting.
Precautionary Backups
A great way to push the speed envelope while reducing your risk of database corruption is the practice of keeping weekly backups. Should a database corruption occur, you can immediately restore UA from your backup copy and then re-collect daily data to fill in the one-week gap. There is no extra charge for doing this. If, on the other hand, you haven’t kept a backup and your database becomes corrupted, you’ll need to get a new UA from CSI. This is available by download for users with a fast connection. Otherwise, you’ll need to order a new CD-ROM. There is a $20 fee for the CD, plus shipping charges if courier or overseas delivery is needed.
To back up your UA, use these instructions, which are very similar to those found in the software manual:
Right-click your computer's Start button, then click "Explore." A screen showing your drive content displays. Scroll through the left window and note two things: 1) the location of your UA folder, and 2) whether or not a backup folder called UAbackup exists.
If there is no UAbackup folder, create one by first clicking your hard drive, which is likely identified as “local disk (c:).” Next click the File menu and, and from there, click NEW and then FOLDER. When prompted, name the folder UAbackup.
Your UAbackup folder will be created and displayed in the left column along with your UA folder. Working in the left window, right-click your UA folder and then, from the resulting menu, click "Copy." Next right-click the UAbackup folder. From the resulting menu, click "Paste." This will create a duplicate copy of your UA folder, including all data stored within its subdirectories, in the UAbackup folder.
Restoring the backup copy is essentially the reverse of the above instructions. If your UA software is corrupted, you’ll copy the UA folder found within the UAbackup folder (by right-clicking it and then clicking “Copy”) and paste it in the root directory of your hard drive (local disk c:). This is done by right-clicking the drive letter and then clicking “Paste.” When asked about replacing files, click [Yes to all]. After restoring your UA directory, you must collect a daily update to bring your database current.
Second Computer Alternative
Our price list shows a small surcharge of 20% for downloading from two computers each day. For our customers who regularly use two computers, you might consider setting one to download with limited acceleration (your backup), and the other for maximum acceleration (most at risk). In the event of a failure, this setup would allow you to use one computer’s database as backup for the other by transferring the UA directory via CD ROM. (See Tech Talk for instructions.) If you stagger the downloads, both computers could be set for maximum accelerations without risk of losing access to a working UA.
Turning DMA On
Depending upon your version of Windows, DMA is either on or off by default. Turning DMA off virtually eliminates database corruption. In testing, have never found database corruption if DMA was turned off, even with UA set at the riskiest setting. Turning DMA on, however, makes a big speed improvement. See Tech Talk in the 1/03 CSI Technical Journal for more information.
UA Speed-Up Tip #2
From the UA Preferences screen, click “Advanced 1.” UA offers a way to improve the speed of making corrections to your database through the “When correcting, keep copy of database in memory,” setting at the bottom of the screen. When UA corrects a database, it makes an uncompressed copy of the database, corrects it, and then recompresses it. If the uncompressed copy is kept in memory (5 MB for futures, 10 MB for stocks, or 60 MB for options), UA can perform these operations much more quickly than if the intermediate copy is stored to the hard drive. If you have an extra, 5, 10, or 60 MB of memory, respectively, available when distributing in UA, you should check this setting. If not, it is probably worth your money to add some extra memory.
UA includes more settings that can be adjusted to speed data distribution, but we’ve run out of space to write about them. Please check the UA online manual for information. At CSI, we are firmly committed to providing all our customers the very best in data services. There is no substitute for quality, and we strive daily to keep you supplied with software and data that go beyond your expectations. My staff and I welcome and appreciate your comments and suggestions.
Bob Pelletier
Tech Talk
Each month in this column the CSI Technical Support Staff addresses topics of interest to many subscribers in a question-and-answer format. This month they discuss what to do in case of software corruption. Information on ordering a new disk, plus tips on creating and using your own backup copies on CD ROM are provided.
Q. My Unfair Advantage software will not run at all. What should I do?
A. It sounds as if your system has somehow become corrupted and must be reinstalled, either from a new disk provided by CSI, or from a recent backup copy. You can order a new disk from CSI by e-mail: techsupport@csidata.com or by calling 561-392-8663. A nominal charge will apply. We recommend you back up your Unfair Advantage system weekly to avoid extra fees and to assure uninterrupted service in case of hardware problems such as this.
Q. How can I back up my Unfair Advantage onto a CD ROM?
A. Here are instructions for backing up your UA database using a Windows XP system with a CD burner:
1) Place a new, recordable compact disk in your CD drive.
2) Next, click the “Start” button and then click the “My computer” icon. Under “Hard Disk Drives,” select the drive that holds your UA database (probably C:\) and double-click it. Next find the folder named “UA” and right-click it. On the new menu that displays, move your cursor to “Send to,” which brings up an additional menu. On that menu, click “CD ROM drive” (or DVD/CD-RW Drive or whatever your writable CD drive is called). It is likely also referred to as D:\. Your computer will begin preparing the data to copy.
3) Return to the “My computer” screen by clicking the “Back” button, and locate your CD ROM drive under “Devices with removable storage.” Double-click your CD ROM drive, and then, in the left-hand panel, click on “Write these file to CD.” A wizard will display to walk you through the rest of the process. Follow the on-screen directions.
4) When the backup is complete, remove and label the CD ROM. Be sure to include the date, and to store it safely.
Q. What if my computer has a CD burner, but not Windows XP?
A. If your computer doesn’t use Windows XP, writing a CD ROM is only slightly more complicated. Instead of going through “My Computer,” you’ll launch your usual CD burning software and follow the instructions to copy the folder named \UA from your data drive onto the CD ROM.
Q. How do I restore the UA system using a backup copy from a CD ROM?
A. Whatever backup method you used, these instructions will allow you to easily restore your UA system.
1) Place your most recent UA backup disc into your CD drive and click your “Start” button. Click “My computer” from the Start menu, which will produce a list of available drives. Scroll as necessary to find your CD ROM drive and then double-click it. When the content displays, right-click the folder called \UA, and from the menu, select “copy”.
2) Next right-click your hard drive (probably C:\). A new menu displays. Click “paste.” You will be asked if you wish to overwrite the existing files; click “yes.” When the copying is finished, you will have successfully replaced your previous installation of Unfair Advantage.
Important Note: Due to the date-sensitive nature of your Unfair Advantage database, no backup copy that is more than four weeks old should be used. Such a backup will not function. We recommend that you back up your UA weekly (either on CD or in another folder on your hard drive) to avoid a lengthy download after restoring the software.
800-274-4727 | 561-392-8663 | 561-392-1379 (Fax)