Managing Reference Data in Enterprise Databases: Binding Corporate Data to the Wider World

Rating:
Summary: Not useful in the real world
Review: The book is good if you are interested in theory only, bu the author seems to not know all that much about database management systems and real applications. On page 201 in the section on Fusing Codes and Descriptions, he claims that needing to know the Code Type is a disadvantage. Not so if each Code Type is a database view and the application programmer would see each code type, and only that code type, through a database view. That's how the real world would deal with this. In addition he deals with indexes on reference data tables from a retrieval perspecive. Since most of these types of tables would fit in less than thre database pages, most SQL optimizers would scan the table and NOT use an index access. He also mentions that performance can be improved by placing all reference data tables in physical proximity on disk. In the real world, reference data tables would be place into their own buffer pool and the physcial access after the first read becomes irrlevent. Not a good read if you need to implement a real world system.

Rating:
Summary: Unique and focused on an often overlooked subject
Review: This unique book fills a much needed void in data architecture and system design. Reference data is too often done as an afterthought when data architecture starts taking form. Worse, it is too often misused, resulting in no clear boundaries between transaction data structures and real reference data.

Mr. Chisholm starts with definitions of what reference data is, and what it is not. He defines reference data as data that is used to categorize, classify, or otherwise qualify or constrain transaction data. In other words, reference data is typically used in look-up tables. This brings us to what reference data is not - it's not transaction structure data, although it is commonly miscast into this role. To see the difference using an example from my own experience, a two column table of reference data could be a digital phone manufacturer and the first three octets of an electronic serial number that is assigned to the manufacturer. This is static. Extending this example to another table of reference data that is keyed to the first could be the manufacturer and specific phone models. While models come and go, this is still static and qualifies as reference data. However, a designer might be tempted to include a table that lists a specific phone, keyed on the electronic serial number, but also keyed to a status, such as active, inactive or stolen. This is a transaction structure because it changes based on a number of conditions, such as if the equipment is active and assigned to a customer, or in stock. Therein lies the distinction and the theme of the book - reference data is static and other types of data are not.

The author covers basic database design concepts, types of reference data and their use, and managing reference data. As the book progresses it is apparent that there is a lot more to consider about reference data than including look-up tables in a database. This is amply shown in the chapter on the reference data life cycle where he discusses changing codes and descriptions. Other issues that the data architect needs to take into consideration, thoroughly covered in this book, are mapping reference data, and implementing reference data in an existing database. Too often look-up tables are added to the schema with no thought to how it is going to be integrated and maintained. The author gives some sage advice on this topic.

I particularly liked the discussion on what I call "reference data from hell" - dates, times & time zones, currency conversions and the such. Also germane because I recently did a consulting assignment in the Middle East is the treatment of foreign languages and translations. I was most gratified, however, to see a chapter that addressed testing reference data and the associated problems with production vs. test data. From personal experience I can attest that this is often overlooked and is a real problem if your data architecture uses a lot of reference data. My main experience is wireless telecom provider billing systems which use a *lot* of reference data. Mr. Chisholm's advice on testing is on the mark and should be heeded.

This book is valuable to any data architect or system designer, however, its real value is to those who are working with e-commerce systems and telecommunications billing systems, because both domains heavily use reference data and need the mature approach set forth in this book.