Download Nearly-Sortedness Test and Its Applications in Databases and more Study notes History of Education in PDF only on Docsity!

Tester for Nearly-Sortedness

and its Applications in Databases

Arie Matsliah

Sagi Ben-Moshe Eldar Fischer Yaron Kanza

Outline

• New definition: “nearly-sorted”
• Tolerant tests for the property of being

nearly-sorted

• Applications in Databases

Example (data)

apartments for rent

Address Fee ($) Size (m^2 ) … Main rd 29, … 500 70 … First ave 3, … 850 120 … … …

records

attributes

Example (query)

query

select all apts. in NY, having size between 70 and 80 m^2 , present sorted by monthly fee

Facts

• Some queries/operations can be processed more efficiently if the data is ordered (sorted acd. to some attribute)
• Additional examples: natural join, intersection, union, except, …
• However, in many cases processor cannot assume the data is ordered

Observation (from experiments)

• Monitoring the “sort” function of a DB-management system (PostgreSQL)
• In many cases, even before sorting the data is “nearly sorted”
• Idea:
  1. test whether the data is “nearly sorted”
  2. if it is – use sorting algorithm that is tailored for nearly-sorted data

Definition: Nearly-Sorted

• f:[n]R
• R – attribute values, total order (<,≤,>,≥)
• f is sorted if for all i<j, f(i)≤f(j)
• f is k-sorted if for all i,j: i≤j-k  f(i)≤f(j)
• 1-sorted  sorted
• f is e-close to being sorted if for some E[n], |E|≤en: f|[n]\E is sorted
• f is (e,k)-nearly-sorted if for some E[n], |E|≤en: f|[n]\E is k-sorted

Example 1

• 1/n-close (e=1/n), n-sorted (k=n)

i

f(i)

Example 3

• (1/5,2)-nearly-sorted

i

f(i)

Next

• (Tolerant) Test for nearly-sortedness
• Algorithm for sorting nearly-sorted

functions

• Experiments

• (i,j) is a k-violation if i≤j-k and f(i)>f(j)

k-Violations and (δ,k)-Active Indices

50 …^5

k

• (i,j) is a k-violation if i≤j-k and f(i)>f(j)
• i is (δ,k)-active if for ≥δ(j-i) indices h[i,j], (i,h) is a k-violation

k-Violations and (δ,k)-Active Indices

50 …^5

k

…

δ(j-i) values < 50

• either i or j must be (½-k/(j-i),k)-active

 if f is not (e,k)-nearly sorted,

(½-k/(j-i),k)-actives ≥ en

k-Violations and (δ,k)-Active Indices

50 …^5

k

…

(j-i)-2k

…

k

Towards tolerant testing based on [ACCL]

Lemma

• if f is (e,k)-nearly sorted then

(1/4,k)-actives ≤ 5 en

• if f is not (6e,6k)-nearly sorted then

(1/3,k)-actives ≥ 6 en