A Query-Driven Characterization of Linked Data

Harry Halpin, <H.Halpin@ed.ac.uk>

Some graph

A Query-Driven Characterization of Linked Data

Problem: Characterizing Linked Data?

There has not been too much work in empirically characterizing the Semantic Web since the Ding and Finin 2006 study, which was before Linked Data and showed that most of the Semantic Web was FOAF profiles and RSS 1.0 feeds.

Lots of large special interest Linked Data, but don't want to completely discount them.

What does the average user want from Linked Data?

Hypothesis: We can sample a hypertext query log and run it against a Linked Data search engine.

Are there URIs in Practice

Method: : Sample the Semantic Web using a query log, prune any query with less than 10 repetitions. Remove everything that seems to be a URI, looking at names in Alexa.

Entities The gazetteer for names was based on a list of names maintained by the Social Security Administration and the gazetteer for place names was based on the gazetteer provided by the Alexandria Digital Library Project, plus some rules for capitalization etc.

High precision, low recall: Best system at MUC-7, around 94% correct (entities) to 98% correct (concept) over 100 random samples.

Concepts: Restrict to single terms with both hypernym and hyponym in WordNet

Data Set: Microsoft Live Search Query Log for 1 month from 2007

Top Entities Queries

Number of Searches (RDF Hits) Entity:
  1. 7,311 (99) david blaine
  2. 2,997 (134) jessica alba
  3. 2,100 (16723) nick
  4. 1,280 (178) michael hayden
  5. 1,098 (10) marcus vick
  6. 1,092 (199) keith urban
  7. 1,015 (43) lane bryant
  8. 990 (55) desmond dekker
  9. 922 (312) jennifer white
  10. 900 (100) clay aiken
  11. 883 (359) bill cosby

Look, a power-law!

Very few empirical studies have been done on Linked Data, and often of the type "Look, it's a power-law!" without any proper statistical tests. See Clauset, Newman et al. paper referenced in my paper for a better method! correlation frequencies

Entity Queries: alpha = 2.31, with long tail behavior starting around a frequency of 17 and a Kolmogorov-Smirnov D-statistic of .0241, indicating a significant good fit.

Concept Queries: The alpha= of the queries for concept queries was calculated to be 2.12, with long tail behavior starting around a frequency of 36 with a Kolmogorov-Smirnov D-statistic of .017.

Top Concept Queries

    Concepts:
  1. 11,383 (10,767) weather
  2. 10,321 (7,777) dictionary
  3. 3,675 (434,333) people
  4. 3,217 (189,115) music
  5. 3,117 (7,196) monster
  6. 2,192 (1,444) autism
  7. 1,468 (149,436) map
  8. 1,198 (17,562) travel
  9. 1,191 (12,067) pregnancy
  10. 1,104 (82,074) news

Hypertext Query Frequency and Linked Data?

Question: Is the amount of Linked Data returned correlated with the popularity of the query?

No: Spearman's rank correlation statistic was the insignificant .0077 ($p > .05), while for concept queries, the correlation was the still insignificant at .0125 ($p > .05$)

correlation frequencies

Top 10 Domain Names: Entities

What URIs did the data come from?

  1. 76,706 dbpedia.org
  2. 2,992 www4.wiwiss.fuberlin.de
  3. 2,848 en.wikipedia.org
  4. 1,264 upload.wikimedia.org
  5. 940 www.liveinternet.ru
  6. 928 www.w3.org
  7. 668 de.wikipedia.org
  8. 630 fr.wikipedia.org
  9. 491 ontoworld.org
  10. 389 sv.wikipedia.org

Top 10 Domain Names: Concepts

  1. 18,919 dbpedia.org
  2. 3,032 www.w3.org
  3. 694 www.cyc.com
  4. 554 bio2rdf.org
  5. 299 truesense.net
  6. 243 www4.wiwiss.fuberlin.de
  7. 171 ontoworld.org
  8. 143 en.wikipedia.org
  9. 132 www.liveinternet.ru
  10. 97 semanticweb.org
Power-law?: The alpha is 1.53, with long tail behavior starting around 175 and a Kolmogorov-Smirnov D-statistic of .1414, indicating insignificant fit for the power-law distribution.

Too Many URIs

There are too many URIs for the same thing probably...

An average of 1,339 URIs (S.D. 8,000) returned per query. That's a lot, but most may obviously be URIs that just mention the term...

return frequency

Looking at Status Codes

The majority of URIs, 51,873 (74%), served RDF via 303 redirection.

200 status codes without 303 redirection still form a substantial fraction (9%) of Semantic Web URIs - URIs just served not as Linked Data.

All hash convention URIs would by default still technically commit a redirect to be served by a 200 status code.

This is only a minority (27%) of those URIs returning a 200 status code.

The rest are likely caused by people serving RDF that does not have the access to the Web server configuration needed to serve RDF using 303 redirection.

Top 10 HTTP Status Codes for crawled URIs

Number Percent Status Code
51,873 73.97% 303
6,061 8.65% 200
4,517 6.44% 404
4,257 6.07% 500
3,147 4.49% 300
246 0.35% 406
20 0.03% 403
4 0.00% 302
3 0.00% 502

Triple Level Analysis

Moving the level of URIs to the level of the triples accessible from the URIs.

Some URIs were inaccessible, this reduced size of sample to 60,972, a reduction of (13%) from the crawled URIs.

The accessible crawled URIs contained 24,074 accessible crawled concept URIs (95% of all crawled concept URIs) and 36,898 (82% of all crawled entity URIs) accessible crawled entity URIs.

Each of the crawled accessible URIs was accessed, and this resulted in a total of 59,228 Semantic Web documents with only 48 URIs not allowing access to a Semantic Web document.

URIs of Vocabulary Terms

Number Percent Vocab URI
366,849 33.55% DBpedia URIs
109,300 9.99% RDF URIs
100,340 9.17% RDF(S) URIs
94,520 8.65% Cyc URIs
34,136 3.12% OWL URIs
6,563 0.60% SKOS URIs
4,728 0.43% dblp.l3s.de URIs
3,263 0.29% FOAF URIS
2,170 0.20% YAGO URIs
1,836 0.16% WordNet URIs

Triple Level Statistics

A total of 411,574 RDF triples in the crawled triples, with 242,829 (59%) triples for concepts and 168,745 (41%) triples for entity URIs.

Of these triples, there were a total of 1,051 triples containing blank nodes, a measly .25% of all triples in the corpus, of which 772 (73%) were subjects and only 279 (27%) were in the object position.

This means that the use of blank nodes are almost non-existent in our sample.

Removing blank nodes, the composition was split between URI nodes (66%) and a surprisingly large minority of RDF literals nodes (34%).

Of the literals, a total of 403,119 (almost 98%) were RDF string literals, while only 2% were of some other data type.

Common Data Types in Crawled Triples

Number Percent Data Type
403,119 97.95% RDF plain literal
3,103 0.75% w3c:/XMLSchema#integer
2,789 0.68% w3c:/XMLSchema#string
1,185 0.29% w3c:/XMLSchema#double
522 0.13% w3c:/XMLSchema#date
248 0.06% w3c:/XMLSchema#float
136 0.03% w3c:/XMLSchema#gYear
65 0.02% w3c:/XMLSchema#gYearMonth
59 0.01% dbpedia:Rank
46 0.01% dbpedia:Dollar
14 0.00% w3c:/XMLSchema#int
9 0.00% dbpedia:Percent

What Language? RDF or OWL?

Of the total 1,093,212 URIs in triples harvested from the crawled accessible URIs, only 243,776 (22%) were from one of the primary W3C Semantic Web knowledge representation languages, either RDF, RDF(S), or OWL.
  1. RDF: 109,300 URIs (45%)
  2. RDF(S): 100,340 URIs (41%)
  3. OWL: 34,136 URIs (14%)
Does not mean OWL is irrelevant, as ontologies constructed with OWL could be deployed to model the concepts and entities employed in `instance' data.

The usage of OWL, RDF(S), and RDF terms does not form a power law. This is because while a few terms vastly dominate, the vast majority of other terms are not used at all.

RDF and OWL Constructs in Crawled Triples

Number Percent Language Construct
73,451 30.31% rdfs:Class
47,044 19.30% rdfs:comment
44,113 18.10% rdfs:subClassOf
8,630 3.54% owl:Ontology
7,256 2.97% rdfs:label
6,618 2.14% rdf:Subject
5,107 2.09% owl:ObjectProperty
3,642 1.49% rdfs:subPropertyOf
1,157 0.47% owl:sameAs
535 0.29% rdfs:range

The Great owl:sameAs Debate

One of the most popular OWL constructs is indeed the controversial owl:sameAs term used to declare equivalence.

One critique holds that it is declaring equivalence between things that are not equivalent, so equivalence is being used too much.

Only 47% of overall Semantic Web modelling term usage, it is far from insignificant, with 1,157 occurrences.

Given the amount of Semantic Web URIs returned by the queries, it appears that the manual discovery and publication of co-referential URIs using owl:sameAs falls far behind the actual growth of Linked Data.

Surprise! Likely owl:sameAs is not being used enough.

Conclusions

The Linked Data Web is full of interesting structured data, but mostly on DBpedia.

Most Linked Data seems to actually conform to the 303 redirect and other recommendations (...or is that mostly DBpedia?)

There is a lot of potential information that users may be interested in! So lots of work needs to be done on ranking linked data.

Yet is this data really relevant to the user's needs? How many are returned in are irrelevant. This is in my SemSearch2009 paper.

How badly biased is this sample by the use of FALCON-S? Can we repeat it using other search engines?

Can we more fairly sample Linked Data while excluding semantic spam or just irrelevant and useless data?

Magic Preview Slide

Results of Relevance Judgements:

Results: Hypertext Semantic Web
Resolved: 197 (98%) 132 (66%)
Unresolved: 3 (2%) 68 (34%)
Top Relevant: 121 (61%) 76 (58%)
Top Non-Relevant: 76 (39%) 56 (42%)

Relevance Results: Hypertext

referent

Results of Querying the Hypertext Web

Relevance Results: Semantic Web

referent

Results of Querying the Semantic Web