NARA has released the bulk contents of it’s Archival Research Catalog on data.gov, in ARC XML format.

Unfortunately, when I say bulk contents, I mean bulk contents; the current distribution contains nearly 20GB of raw data; the Organizational authority file adds another 1.7GB.

Since XML was deliberately designed for redundancy, XML encoded data sets are usually rather compressible. These data are no exception; bzip2 on the ARC data set achieves compression rations  between 40:1 and 50:1.   The 7-zip format does even better, but is not considered archival.

The real credit for this work belongs to the good people at NARA and their contributing agencies (and viewers like you). Don’t forget to visit the original pages on data.gov to give comments and feedback on this data, and don’t forget to show off all the cool UIs you build on top of this.

Mark Matienzo, (now a Yalie as of  Monday),  Richard Urban,  GSLIS, University of Illinois,and Simon Spero, SILS, University of North Carolina have prepared a compressed distribution of these data sets, allowing much more efficient access to the data.

This distribution is available from iBiblio at this location; http://www.ibiblio.org/fred2.0/NARA/

One interesting datum that emerged during the compression process is that the compressed size of Part1 is nearly twice as large as most of the other Parts, despite having the same raw size.  There are several possible explanations for this:

• H1: The records in Part1 are more complete than in other files.
• H2: Part1 describes record series from less related agencies than other files.
• H3: Part1 contains records with less compressible data (image types, numeric data, etc).
• H4: Part1 contains more, smaller, record series than other files.

If  only had a time machine, I could do a poster and go to the iConference.  Alas, all I have is the scarf and the Jelly babies, and I’ve eaten the Jelly babies.

This release is dedicated to the memory of Prof. Brian Vickery (1918-2009), whose passing was announced on Thursday the 28th.

Higgledy-Piggledy
Brian C. Vickery
Ranganathoricly
Classified Life

Polyhierarchically
Never too snarkaly
Faceted artfully
Helped refind science

—–

See the Readme file for a bit more details.

It’s time to try and find a latex -> wordpress plugin that isn’t too smart for my own good. I’d like to be able to use some AMS macros, but the plugins seem to be trying to protect me from myself.

Single equations are easy. \text{} works, so there’s some amsmath in there.

$$\text{\sf Unicorns} \sqsubseteq \text{\sf Horses}\\ \text{Horses} \sqcap \text{Regicides} \not\equiv \bot$$

But getting displays to break paragraphs and  getting align to work at all is a whole  different kettle of ballgames.

Those cynical chaps at the Speculative Grammarian make a mockery of all that is good and holy about Computational Linguistics and Information Retrieval.

[...]

So, rather than trying to dumb things down and arrive at such a single “accuracy” number, we propose instead to dumb things up—constructing measures that focus on the real needs of a measurable theory, including the meta-system/contextual-matrix in which it is embedded (including, explicitly and for the first time, the researchers and grad-students on the research team).

These two new measures are called recision and precall.

Recision is a measure of the amount of data that must be ignored (or surreptitiously dumped in the river with a new pair of cement shoes) in order to get publishable results. If 10% of your data must be “lost” in order to get good results that support your pre-computed conclusions, then your theory and your research team have a respectable recision score of 10%. If only 10% of your data is useable, then your recision score is a dismal4 90%.

Precall is a measure of your team’s ability to quickly and correctly predict how well your algorithm or system will perform on a new data set that you can briefly review. Correctly predicting “This will give good results.” or “This is gonna suck!” 90% of the time translates directly into a precall score of 90%. Good precall (especially during live demos) can save a project when results are poorer than they should be. The ability to look at some data and accurately predict and, more importantly, explain why such data will give poor results shows a deep understanding of the problem space.5 Even when performance is decent, though, prefacing each data run with “We have no idea how this will turn out!” makes your team look lucky, at best, or, at worst, foolish.

Remove the following relationship  types:

1. Broader Term (BT)
2. Narrower Term (NT)
3. Related Term (RT)
4. Use For (UF)
5. Use (USE)

Add the following relationship types:

1. It’s Complicated

Advatages

1. It’s Simple

Disadvantages:

1. It’s Complicated

There’s all sorts of wisdom in academia.  It seems to have started in business schools, but LIS has taken up the banner too.    It’s only a matter of time before we start awarding MSWS degrees.

It’s in handbooks of Measuring System Design. This isn’t on UNC’s subscription, so I couldn’t say what the SI unit of Wisdom is, but I’ll take a guess at the kilowit.

Other brave souls have attempted to identify the definitions of the Wisdom hierarchy (and why it changes to the Information Hierarchy when Information Scientists talks about it, and the Knowledge Hierarchy when Knowledge Management professionals are involved).

See The wisdom hierarchy: representations of the DIKW hierarchy for one effort.

After revisiting Ackoff’s original articulation of the hierarchy, definitions of data, information, knowledge and wisdom as articulated in recent textbooks in information systems and knowledge management are reviewed and assessed, in pursuit of a consensus on definitions and transformation processes

We should have plenty of time afterwards for tea and cake.
The sad thing is, I think there is an operationalisable definition available using Shannon’s Information Theory as a starting point, but the result isn’t that exciting.   If you treat Knowledge as the ability to reason backwards from a desired end state from the inferred current state (based on perceived data), then executing that plan, updating current state as more information is learned;  then Wisdom can be the ability to infer forwards to the fuller set of consequences of those actions.  Since the only real law in this area is the law of unintended consequences, this isn’t too useful a definition.

“Data organization.” “Human Information  Organizing Behavior.” “Knowledge Organization Systems.”  What would characterize a “Wisdom Organization  System?”

Self Archiving has failed. Stealth Archiving is the answer.

An ordinary person spends there time getting out avoiding tense situations. A repo-rat spends his time getting in to tense situations.

Never broke into an office. Never wire-tapped an office. Kid. I never broke into a laptop. I shall not corrupt a file northe contents thereof. Nor through inaction let that file or the digital contents thereof come to harm. That’s what I call the repo code kid. Don’t forget it etch it in your brain. Not many people got a code to live by anymore.

TThere’s a critical difference between subject based knowledge organization schemes created to organize books and other documents, (e.g. LCC, DDC, and LCSH), which are defined in terms of the things which a work is about, and rigorous, class based ontologies that are based on the properties of the things themselves.

Some relationships may be valid in terms of subjects, but may not apply directly to the things themselves. For example, a car wheels are part of cars, and cars are a type of vehicle. These relationships are always* true, and are examples of two different types of hierarchical relationships – partitive (is-part-of) (BTP) , and generic (is-a-kind-of) (BTG)

Always-aboutness is the combination of all the different types of hierarchical relationship – this is the Broader Term (BT) relationship. In terms of aboutness, we can easily say that everything about car wheels is always about cars, and everything about cars is always about vehicles. We can also directly infer that everything about car wheels must always be about vehicles. In other words, the always-aboutness relationship must be transitive.

Transitivity doesn’t hold when you mix different types of hierarchy – car wheels are not a kind of vehicle, but that has nothing to do with the transitivity of plain unqualified Broader Term.

When thinking about systems for organizing information, it’s absolutely critical to remember the difference between the document, and the thing the document is about.

Or in other words: