<![CDATA[
<BODY TEXT="black" BGCOLOR="white" LINK="#006666" ALINK="#003333" VLINK="#009999" background="../../images/bk_teal_corp.gif">
<TABLE BORDER="0" CELLPADDING="0" CELLSPACING="0" WIDTH="607">
<TR>
<TD WIDTH="146" VALIGN="TOP">
<TABLE BORDER="0" CELLPADDING="0" CELLSPACING="0" WIDTH="138">
<TR>
<TD COLSPAN="2"><IMG SRC="../../images/clearpixel.gif" WIDTH="12" HEIGHT="10" ALIGN="BOTTOM" BORDER="0"></TD>
</TR>
<TR>
<TD COLSPAN="2"><A HREF="../attitudes.html" target="_top"><IMG SRC="../../images/but_com_att0.gif" WIDTH="138" HEIGHT="20" ALIGN="BOTTOM"
ALT="Attitudes" BORDER="0"></A></TD>
</TR>
<TR>
<TD COLSPAN="2"><A HREF="../news.html" target="_top"><IMG SRC="../images/but_com_news1.gif" WIDTH="138" HEIGHT="25" ALIGN="BOTTOM" ALT="News at Daman"
BORDER="0"></A></TD>
</TR>
<TR>
<TD WIDTH="17" HEIGHT="6"><IMG SRC="../../images/clearpixel.gif" WIDTH="1" HEIGHT="1" ALIGN="BOTTOM" BORDER="0"></TD>
<TD WIDTH="121" HEIGHT="6"><IMG SRC="../../images/clearpixel.gif" WIDTH="1" HEIGHT="1" ALIGN="BOTTOM" BORDER="0"></TD>
</TR>
<TR>
<TD WIDTH="17"> </TD>
<TD WIDTH="121"><A HREF="../projects.html" target="_top"><FONT SIZE="1" FACE="Verdana, Helvetica, Arial">Current Projects</FONT></A></TD>
</TR>
<TR>
<TD WIDTH="17" HEIGHT="6"><IMG SRC="../../images/clearpixel.gif" WIDTH="1" HEIGHT="1" ALIGN="BOTTOM" BORDER="0"></TD>
<TD WIDTH="121" HEIGHT="6"><IMG SRC="../../images/clearpixel.gif" WIDTH="1" HEIGHT="1" ALIGN="BOTTOM" BORDER="0"></TD>
</TR>
<TR>
<TD WIDTH="17" HEIGHT="22"> </TD>
<TD WIDTH="121" HEIGHT="22"><A HREF="../press_rel.html" target="_top"><FONT SIZE="1" FACE="Verdana, Helvetica, Arial">Press Releases</FONT></A></TD>
</TR>
<TR>
<TD WIDTH="17" HEIGHT="6"><IMG SRC="../../images/clearpixel.gif" WIDTH="1" HEIGHT="1" ALIGN="BOTTOM" BORDER="0"></TD>
<TD WIDTH="121" HEIGHT="6"><IMG SRC="../../images/clearpixel.gif" WIDTH="1" HEIGHT="1" ALIGN="BOTTOM" BORDER="0"></TD>
</TR>
<TR>
<TD WIDTH="17"> </TD>
<TD WIDTH="121"><A HREF="../headlines.htm" target="_top"><FONT SIZE="1" FACE="Verdana, Arial, Helvetica">News About Daman</FONT></A></TD>
</TR>
<TR>
<TD WIDTH="17" HEIGHT="6"><IMG SRC="../../images/clearpixel.gif" WIDTH="1" HEIGHT="1" ALIGN="BOTTOM" BORDER="0"></TD>
<TD WIDTH="121" HEIGHT="6"><IMG SRC="../../images/clearpixel.gif" WIDTH="1" HEIGHT="1" ALIGN="BOTTOM" BORDER="0"></TD>
</TR>
<TR>
<TD WIDTH="17"><IMG SRC="../../images/arrw_teal_corp.gif" WIDTH="17" HEIGHT="10" ALIGN="BOTTOM" BORDER="0"></TD>
<TD WIDTH="121"><A HREF="../artcls.htm" target="_top"><FONT SIZE="1" FACE="Verdana, Helvetica, Arial">Articles & White Papers</FONT></A></TD>
</TR>
<TR>
<TD WIDTH="17" HEIGHT="6"><IMG SRC="../../images/clearpixel.gif" WIDTH="1" HEIGHT="1" ALIGN="BOTTOM" BORDER="0"></TD>
<TD WIDTH="121" HEIGHT="6"><IMG SRC="../../images/clearpixel.gif" WIDTH="1" HEIGHT="1" ALIGN="BOTTOM" BORDER="0"></TD>
</TR>
<TR>
<TD COLSPAN="2"><A HREF="../events.html" target="_top"><IMG SRC="../../images/but_com_eve0.gif" WIDTH="138" HEIGHT="20" ALIGN="BOTTOM"
ALT="Events" BORDER="0"></A></TD>
</TR>
<TR>
<TD COLSPAN="2"><A HREF="../../contact.html" target="_top"><IMG SRC="../../images/but_com_con0.gif" WIDTH="138" HEIGHT="20" ALIGN="BOTTOM"
ALT="Contact Info" BORDER="0"></A></TD>
</TR>
<TR>
<TD WIDTH="17"><IMG SRC="../../images/clearpixel.gif" WIDTH="17" HEIGHT="1" ALIGN="BOTTOM" BORDER="0"></TD>
<TD WIDTH="121"><IMG SRC="../../images/clearpixel.gif" WIDTH="121" HEIGHT="1" ALIGN="BOTTOM" BORDER="0"></TD>
</TR>
</TABLE>
<DIV ALIGN="RIGHT">
<P>
</DIV>
</TD>
<TD WIDTH="12"> </TD>
<TD VALIGN="TOP" COLSPAN="3">
<p><b><font face="Verdana, Arial, Helvetica" color="#006666">Real-Time Data Warehousing Defined</font></b></p>
<p><b><font size="1" face="Verdana, Arial, Helvetica" color="#006666">By Michael Haisten<br>
Originally published in <a href="http://dmreview.com/master.cfm?NavID=55&EdID=1522" target="_blank">DM Review Online, October, 1999</a><br>
</font></b></p>
<p><font size="1" color="black" face="Verdana, Arial, Helvetica">We are now poised for the most radical transition in the short history of data warehousing. We are about to witness the arrival of real-time data warehousing (RTDW). Despite its name, the impact of this change is due less to the increase in speed of availability (which is significant) than to a significant reduction in overall processing complexity. We will replace a brute force process with elegant, yet powerful, simplicity.</font></p>
<p><font size="1" color="black" face="Verdana, Arial, Helvetica">Real-time data warehousing will eliminate the artificial bottlenecks we have designed into our solutions from the very beginning. It will solve some of the most vexing data warehouse problems such as change data capture. It will allow us to return to co-engineered, end-to-end system architectures.</font></p>
<p><font size="1" color="black" face="Verdana, Arial, Helvetica">So what is RTDW? It is a historical and analytic component of an enterprise-level data stream. This data stream supports continuous, asynchronous, multipoint delivery of data. In other words, data moves straight from the originating source to all uses that do not require some form of staging. This movement takes place soon after the original data is written. Any time delays are due solely to transport latency and (optionally) minuscule processing times to dispatch or transform the instance of data being delivered.</font></p>
<p><font size="1" color="black" face="Verdana, Arial, Helvetica">The data warehouse component of this information delivery architecture contains both a real-time and a static partition. The real-time partition gets its data an instance at a time from a transformation engine. At regular intervals, consistent subsets of the real- time data are written to the static partition. The static partition maintains the history and serves as the base for supplying dependent data marts with periodic data. </font></p>
<p><font face="Verdana,Helvetica,Arial" size="1"><b><a href="sidebar_1099.html" target="_blank">Diagram and Sidebar: Components of a Real-Time Data Distribution Environment </a></b></font></p>
<p><b><font size="1" face="Verdana, Arial, Helvetica" color="#006666">Traditional Data Warehousing as Archaeology<br>
</font></b><font size="1" color="black" face="Verdana, Arial, Helvetica">Building a data warehouse is like an archaeological expedition. You generally know what you are looking for, but you have to decipher dusty old records to find where to dig. You set up elaborate scaffolding that can only be used once to get at the buried artifacts, and the work is always dirty. When you find the old relics, you must carefully package them for transport to a laboratory where they are unwrapped and reassembled for display. This is all massively labor-intensive; and, in the end, you only have a bunch of old stuff that represents an imperfect subset of what actually happened in the past.</font></p>
<p><font size="1" color="black" face="Verdana, Arial, Helvetica">Why is the record incomplete? A flood may wash away a whole layer of the fossil record. This is similar to a purge or selective but undocumented periodic deletions. A geological intrusion can replace something without leaving evidence of what was there before. Our operational systems allow the update of information that effectively destroys the historical record. The items that survive are those that were built to last or were intentionally preserved based on the cultural preferences and bias of that time. Since it is very expensive to store things to last, few things survive. Likewise with information systems, we have been very selective about what we keep since we generally view storage as a scarce and costly luxury.</font></p>
<p><font face="Verdana,Helvetica,Arial" size="1" color="#006666"><b>Perfect Capture</b></font><br>
<font size="1" color="black" face="Verdana, Arial, Helvetica">First and foremost, real-time data warehousing eliminates the dig. The data we need now and the data we will need in the future streams out of the sources continuously as they come into being. The vast majority of the labor-intensive development effort, the 65 percent to 80 percent involved with extraction, is eliminated.</font></p>
<p><font size="1" color="black" face="Verdana, Arial, Helvetica">Changed data capture is the process of collecting incremental net changes so we can record history accurately. It allows us to maintain point-in-time currency. It is essential to doing data warehousing right; but it is often very difficult, if not impossible, to do after the fact by reading operational data sources in batch. In RTDW, a perfect form of changed data capture is available at no extra cost. Both the before and the after image of a record is available in memory supporting easy and efficient delta processing. For dimensional data, you have much more freedom in how you represent slowly changing dimensions.</font></p>
<p><font size="1" color="black" face="Verdana, Arial, Helvetica">When an event occurs, such as shipping an order, all relevant information can be captured simultaneously. Today, if we get an extract from a shipping subsystem, it is not likely to have relevant customer, product, order or invoice cross-references necessary to fully describe the event. We have to piece them together by merging data from multiple sources to artificially recreate the context of the event. Differing update timing may make some of the associations spurious. There is enormous advantage in being able to capture the full context of an event at creation time.</font></p>
<p><font size="1" face="Verdana, Arial, Helvetica" color="#006666"><b>Quick Reconfiguration</b></font><font size="1" color="black" face="Verdana, Arial, Helvetica"><br>
By definition, our systems record only what we tell them to record. Even the monolithic enterprise resource planning suites cannot be designed to anticipate everything we may need to analyze or record for posterity. The ideal capture mechanism for a RTDW is an application service that uses templates (such as SAP’s IDOC) that specify a data context with an associated trigger event</font><font size="1" color="black" face="Verdana, Arial, Helvetica">.</font></p>
<p><font size="1" color="black" face="Verdana, Arial, Helvetica">Such a template-driven mechanism allows the data warehouse designer to reconfigure an existing capture specification to include more content in hours. A wholly new specification can be ready for implementation in a matter of days. The limiting factor for availability is likely to be the turnaround time for production migration rather than development time. This is in contrast to a minimum of three to five days for building a single-purpose interface when using an extraction, transformation and loading (ETL) tool or the two to three weeks when writing it by hand.</font></p>
<p><font size="1" face="Verdana, Arial, Helvetica" color="#006666"><b>Time Compression</b></font><font size="1" color="black" face="Verdana, Arial, Helvetica"><br>
As the pace of business picks up, processing cycles get compressed. Buying via mail order used to take a week or two. Express delivery services helped reduce this to two or three days. Now when you buy by phone or on the Internet, you can have your product the very next day. This means the product is delivered to the shipper the day the order is received.</font></p>
<p><font size="1" color="black" face="Verdana, Arial, Helvetica">Keeping pace with this speedy processing puts a strain on the traditional distinctions between operational and analytic information services. We use trended data regarding what sells well by site by hour to manage daily decisions. Actual daily sales activity is monitored to replan just- in-time inventory replenishment that, in turn, impacts longer cycle purchase planning.</font></p>
<p><font size="1" color="black" face="Verdana, Arial, Helvetica">A real-time acquisition and delivery engine enables on-the- fly analysis while ratcheting up seamlessly to support incremental aggregation and trend-line confirmation. The goal is to support immediate research of abnormal conditions in a manner not supported by the OLTP system. For example, a real-time transportation system getting satellite fed data of carrier deliveries indicates that an abnormal number of shipments is unloading behind schedule. On-the-fly analysis reveals the slow handling is for customer self- unloads and that the affected shipments are a brand-new multiproduct configuration. This information is used to quickly write and transmit new breakdown instructions to these customer sites.</font></p>
<p><font size="1" color="black" face="Verdana, Arial, Helvetica">Incremental aggregation is an alternative to the old method of creating running totals. At regular intervals, such as every 15 minutes for securities or every 30 to 60 minutes for product orders or bank transactions, an aggregation engine runs against the real-time partition to create intermediate totals and statistics. These aggregates can be cascaded into long-period aggregates (such as daily) and then stored in the static partition of the data warehouse. Because this aggregation is done more continuously, it can actually support higher volumes than more traditional daily or weekly bulk processing.</font></p>
<p><font size="1" color="black" face="Verdana, Arial, Helvetica">Trend-line confirmation builds on incremental aggregation to provide a new monitoring tool. Any time- sensitive activity, such as dropped calls at cells sites, can be compared to a baseline at close to real time. This provides much more insight and control than even active "stop lighting," which only tells you when cell sites have already overloaded. Trend-line confirmation reveals the momentum and direction of a change that is out of the ordinary soon after it begins.</font></p>
<p><font size="1" face="Verdana, Arial, Helvetica" color="#006666"><b>Transitory States</b></font><font size="1" color="black" face="Verdana, Arial, Helvetica"><br>
Even if your enterprise is not (yet) running near real time, you may be prone to the effect of time compression. A key business process may be impacted by transitory states. These are short duration activities that may end in some form of reversal that leaves little or no evidence behind.</font></p>
<p><font size="1" color="black" face="Verdana, Arial, Helvetica">For instance, a credit hold is a temporary condition in an online order-taking system. An order goes on hold and then comes off, either with the order approved or disapproved. For an approved order, you may not know that it went on hold at all. At most, there is a residual hold indicator in the system. Disapproved orders will generally record a credit rejection reason. But, in neither case are the details of the hold event recorded. What happens when you introduce a new product that appeals to a consumer segment with lower credit worthiness? The number of orders per representative may decline drastically because of hold delays, but you cannot see this effect directly. The number of disapproved orders may increase, but this does not account for the lower number of orders per rep.</font></p>
<p><font size="1" color="black" face="Verdana, Arial, Helvetica">With a real-time acquisition mechanism, you can capture the on-hold and off-hold events as they occur. Due to the ability to quickly reconfigure the data stream to capture new events, you can begin analysis of a new phenomenon soon after you first notice it. Combined with other state transition events, you can create a contemporaneous record that can be used for analyzing the full order cycle.</font></p>
<p><b><font size="1" face="Verdana, Arial, Helvetica" color="#006666">Infinite Snapshots</font></b><font size="1" color="black" face="Verdana, Arial, Helvetica"><br>
Many data warehouse traditionalists will already be planning their rebuttals. Real time and warehousing just do not mix! Warehouses are about snapshots that provide a consistent point-in-time view of the data. They argue that real-time data capture violates this treasured principle.</font></p>
<p><font size="1" color="black" face="Verdana, Arial, Helvetica">The enlightened perspective is that you cannot achieve point-in-time consistency without continuous, real-time data capture. The point-in-time that is relevant for one use may be radically wrong for another. Some uses may need to count orders taken but not released, released but not shipped, invoiced but not paid, etc. The most flexible data design does not replicate the document concepts of OLTP system. Instead it captures information about discrete states in a business process as they occur.</font></p>
<p><font size="1" color="black" face="Verdana, Arial, Helvetica">With real-time and cross-application data capture, we are finally free to implement designs based on event states rather than documents or transactions. Continuing the order example, each row in a state-based table is a snapshot of what is known about the order when it reached that state (taken, picked or shipped). In a sense, a state table is a far superior "snapshot," since every row in it represents what is known about the order at a consistent state in the process.</font></p>
<p><font size="1" color="black" face="Verdana, Arial, Helvetica">Inventories and balances are a unique category of snapshot. They record the quantity or level of some tangible asset as of a particular valuation point. Whether they are numbers that reflect actual inventory counts or are derived by netting in-flows and out- gos, they are defined by the discrete time of their calculation. Today, the number of relevant "as-of" times is increasing. You might have month end immediate for one use, month end after the first weekend of processing for another and month end final after all adjustments. Weekly, daily, shift and even instant balances now complement these long-cycle views.</font></p>
<p><font size="1" color="black" face="Verdana, Arial, Helvetica">The best, and possibly only, way to account for all the possible variations involves continuous sampling using real-time techniques. Any balance required for legal, analytic or other reason needs to be frozen for posterity. We do this by writing it to the static data warehouse partition. A data warehouse must be designed to collect a large number of snapshots based on the consistency goals of the driving need.</font></p>
<p><font size="1" color="black" face="Verdana, Arial, Helvetica">The technologies to support real-time data warehousing are coming online today. The early adopters are already developing the techniques. The ball is in now in your court.</font></p>
</TD>
</TR>
<TR>
<TD WIDTH="146"><IMG SRC="../../images/clearpixel.gif" WIDTH="138" HEIGHT="6" ALIGN="BOTTOM" BORDER="0"></TD>
<TD WIDTH="12"><IMG SRC="../../images/clearpixel.gif" WIDTH="12" HEIGHT="5" ALIGN="BOTTOM" BORDER="0"></TD>
<TD WIDTH="309"><IMG SRC="../../images/clearpixel.gif" WIDTH="309" HEIGHT="6" ALIGN="BOTTOM" BORDER="0"></TD>
<TD><IMG SRC="../../images/clearpixel.gif" WIDTH="15" HEIGHT="5" ALIGN="BOTTOM" BORDER="0"></TD>
<TD WIDTH="125"><IMG SRC="../../images/clearpixel.gif" WIDTH="125" HEIGHT="5" ALIGN="BOTTOM" BORDER="0"></TD>
</TR>
</TABLE>
<TABLE BORDER="0" CELLPADDING="0" CELLSPACING="0" WIDTH="608">
<TR>
<TD WIDTH="25"> </TD>
<TD COLSPAN="3">
<CENTER>
<P>
<HR ALIGN="CENTER" SIZE="1" NOSHADE>
<BR>
<A HREF="../../index.html" target="_top"><FONT SIZE="1" FACE="Verdana, Helvetica, Arial">home</FONT></A><FONT SIZE="1"
FACE="Verdana, Helvetica, Arial"> | </FONT><A HREF="../../aboutus.html" target="_top"><FONT SIZE="1" FACE="Verdana, Helvetica, Arial">about
us</FONT></A><FONT SIZE="1" FACE="Verdana, Helvetica, Arial"> | </FONT><A HREF="../../solutions.html" target="_top"><FONT
SIZE="1" FACE="Verdana, Helvetica, Arial">solutions</FONT></A><FONT SIZE="1" FACE="Verdana, Helvetica, Arial">
| </FONT><A HREF="../../products.html" target="_top"><FONT SIZE="1" FACE="Verdana, Helvetica, Arial">products</FONT></A><FONT
SIZE="1" FACE="Verdana, Helvetica, Arial"> | </FONT><A HREF="../../partners.html" target="_top"><FONT SIZE="1" FACE="Verdana, Helvetica, Arial">partners</FONT></A><FONT
SIZE="1" FACE="Verdana, Helvetica, Arial"> | </FONT><A HREF="../../clients.html" target="_top"><FONT SIZE="1" FACE="Verdana, Helvetica, Arial">clients</FONT></A><FONT
SIZE="1" FACE="Verdana, Helvetica, Arial"> | </FONT><A HREF="../../careers/home.html" target="_top"><FONT SIZE="1" FACE="Verdana, Helvetica, Arial">careers</FONT></A><FONT
SIZE="1" FACE="Verdana, Helvetica, Arial"> | </FONT><A HREF="../../contact.html" target="_top"><FONT SIZE="1" FACE="Verdana, Helvetica, Arial">contact
us</FONT></A><FONT SIZE="1" FACE="Verdana, Helvetica, Arial"> | </FONT><A HREF="../../search.htm" target="_top"><FONT
SIZE="1" FACE="Verdana, Helvetica, Arial">search</FONT></A><A HREF="../../sitemap.html" target="_top"><FONT SIZE="1"
FACE="Verdana, Helvetica, Arial"><BR>
</FONT></A><FONT SIZE="1" FACE="Verdana, Helvetica, Arial"><IMG SRC="../../images/clearpixel.gif" WIDTH="1" HEIGHT="10"
ALIGN="BOTTOM" BORDER="0"></FONT></CENTER>
</TD>
</TR>
<TR>
<TD WIDTH="25" VALIGN="BOTTOM"> </TD>
<TD WIDTH="121" VALIGN="TOP">
<P ALIGN="RIGHT"><IMG SRC="../../images/logo_sm.gif" WIDTH="120" HEIGHT="40" ALIGN="BOTTOM" BORDER="0">
</TD>
<TD WIDTH="12"> </TD>
FACE="Verdana, Helvetica, Arial">Texas</FONT><FONT SIZE="1" COLOR="white" FACE="Verdana, Helvetica, Arial">.</FONT><FONT
SIZE="1" COLOR="#666666" FACE="Verdana, Helvetica, Arial">78746 tel(512)329-6646,</FONT><FONT
SIZE="1" COLOR="white" FACE="Verdana, Helvetica, Arial">.</FONT><FONT SIZE="1" COLOR="#666666" FACE="Verdana, Helvetica, Arial">fax(512)329-0767<BR>
<IMG SRC="../../images/clearpixel.gif" WIDTH="12" HEIGHT="5" ALIGN="BOTTOM" BORDER="0"><BR>
</FONT><I><FONT SIZE="1" COLOR="#999999" FACE="Verdana, Helvetica, Arial">© Copyright 1999-2003, Daman Incorporated. All rights reserved. All trademarks are the property of their respective owners.</FONT></I></TD>
</TR>
<TR>
<TD WIDTH="25"><IMG SRC="../../images/clearpixel.gif" WIDTH="25" HEIGHT="1" ALIGN="BOTTOM" BORDER="0"></TD>
<TD WIDTH="121"><IMG SRC="../../images/clearpixel.gif" WIDTH="121" HEIGHT="1" ALIGN="BOTTOM" BORDER="0"></TD>
<TD WIDTH="12"><IMG SRC="../../images/clearpixel.gif" WIDTH="12" HEIGHT="1" ALIGN="BOTTOM" BORDER="0"></TD>
<TD WIDTH="450"><IMG SRC="../../images/clearpixel.gif" WIDTH="450" HEIGHT="1" ALIGN="BOTTOM" BORDER="0"></TD>
</TR>
</TABLE>
<P><B><A HREF="http://www.iemagazine.com/990903/feat2.shtml" target="_blank"><SPAN
STYLE="Font-Family : Verdana, Arial, Helvetica ; Font-Size : XX-Small"></SPAN></A></B></P>
<P><A HREF="../solutions/data_org/white_paper.htm" target="_top"><SPAN STYLE="Font-Family : Verdana, Arial, Helvetica ; Font-Size : XX-Small"></SPAN></A></P>
</BODY>
]]>