Albert-Ludwigs-Universit¨ at, Inst. f¨ ur Informatik Prof. Dr. Fabian Kuhn S. Daum, H. Ghodselahi, O. Saukh December 5, 2013 Algorithm Theory Wintersemester 2013 Problem set 4 – Sample solution Exercise 1: Binomial Queue We have seen in the lecture that when using a Binomial heap to implement Dijkstra’s algorithm for a graph G with n nodes and m edges, we can upper bound the total running time by O(m log n). The goal of this exercise is to show that this bound is tight for dense graphs. Give a graph G = (V, E) with positive edge weights and m = n2 = Θ(n2 ) edges on which Dijkstra’s algorithm implemented with a Binomial heap requires Θ(n2 log n) time. When solving the exercise, you can assume that initially the nodes are inserted into the heap in the worst possible order. You still need to show the inefficiency of using that heap in your example. Solution Since we already know that all operations in a Binomial heap take at most O(log n) time steps we need to force Dijkstra to do expensive operations (decrease-key) for each edge in the graph (or almost all edges). We start out with constructing a weighted graph that, if Dijkstra is executed at one specific node, needs indeed m = Θ(n2 ) decrease-key operations. After that we show that all those decrease-key operations are costly in average. We need a graph such that while we go from node to node (with Dijkstra), every step we realize that the current node is closer to all unvisited nodes in the graph. That can be achieved simply as follows. Start with node v1 and let it have distance D to all nodes but v2 , to which it has distance . So we next visit v2 and to decrease all other nodes’ keys, we need the distance from v2 to all other nodes to be something strictly smaller than D − , say D − 2; except to the next node v3 , which has again distance . For v3 the new distances need to be strictly smaller than D − 3, say D − 4, and so forth. To ease notation we deduct another 2 from all long distances and get the following distance scheme for i < j: ( j =i+1 d(vi , vj ) = D − 2i j > i + 1 With < D/2n we get a graph such that if we start Dijkstra at node v1 , after each step we need to decrease the keys of all nodes. This is already sufficient to make the heap fail, but we adjust our distance function later to make it even more obvious. Now we look at a 12-node heap directly after initialization. 1 v9 v8 v7 ? ? v2 v3 ? v5 v4 ? v6 We can assume an arbitrarily bad starting heap, so we put v2 to v5 in the smaller tree. Dijkstra does not make any statement about in which order it decreases the keys, so let’s assume it starts with the lowest indexed node. For the moment ignore what happens in the smaller tree and focus on the bigger tree. Decreasing the key of v6 means that it swaps positions with all its parents until the heap property is restored. This will move it to the root position. The initial parent of v6 is now the farthest from the root, so let’s decide this to be v7 (remember that we can start with an arbitrary heap). v7 travels almost all the way to the root and then it stops. We would prefer v7 however to go further and we can do so if we reduce the key of v7 not to D − 2 but to D − 2 − γ1,7 , i.e., the edge from v1 to v7 is slightly shorter than all the other edges from v1 – for this not to interfere with our plan of having all nodes’ keys being decreased in each step of Dijkstra, we need γ1,7 to be smaller than . We also do this for v8 and v9 , with γ1,9 > γ1,8 > γ1,7 , but all much smaller than . Now we can continue this game for all nodes that are far away from the root in that particular binomial tree. For all those nodes we have to do swaps of roughly their depth in the tree – the depth they have when it’s their turn to get their key decreased. Consider one such binomial tree Bk . At least half of all nodes are at distance bk/2c or more from the root. To see this consider an even k and some k level j. As said in the lecture, there are kj nodes in level j – as are in level k − j, since kj = k−j . Thus, level k/2 (roughly) is “mirroring” the number of nodes in the levels on each side. The rest of the argument is trivial. We can order the tree as we want and with the right choices of γi,j we can make all those nodes swap until they reach the root, which costs O(k) steps. Thus, for one iteration of Dijkstra we can have costs as high as Ω(k2k ). However, after each step the binomial heap decreases by one in its size. How does that affect our analysis? Too see this quickly, look at the picture above: removing v2 to v5 will not affect the larger tree, but with the right choices of γi,j we can let all nodes move a lot in the larger tree, as we have already seen. More precisely we show that the first n/2 iterations of Dijkstra cause n/4 of all nodes to have a costly decrease-key operation, where costly means Θ(log n). Take n = 2k − 1 for some k. The heap consists of log n = k trees B0 , B1 , . . . , Bk−1 . We place v2 in the B0 , v3 and v4 into the B1 and so forth. Thus, the first (n − 1)/2 iterations, no node is removed from Bk−1 , just the ordering in that tree is changed. Since Dijkstra accesses and decreases a node’s key in a deterministic way, using the γi,j we can make sure that with each iteration of Dijkstra (we only consider the first n/2 − 1 operations), all nodes in Bk−1 of depth k/2 or more (of which there are at least |Bk−1 |/2 = n/4) are moving all the way up to the root, i.e., decreasing their key costs at least k/2 − 1 = Ω(log n) time. Which totals to O(n2 log n). Note: Using the γi,j weights we can trick any deterministic execution (of which node’s key to decrease first) given by the Dijkstra. However, even if Dijkstra makes random choices, using random (small) weights γi,j will cause a costly heap update with high probability. Moreover, having Dijkstra making random choices is not trivially cheap itself. Exercise 2: Fibonacci Heap (a) Consider the following Fibonacci heap (black nodes are marked, white nodes are unmarked). How does the given Fibonacci heap look after a decrease-key(v, 2) operation and how does it look after 2 a subsequent delete-min operation? 5 8 v 14 17 1 11 20 25 9 12 19 3 31 6 10 7 18 (b) Fibonacci heaps are only efficient in an amortized sense. The time to execute a single, individual operation can be large. Show that in the worst case, both the delete-min and the decrease-key operations can require time Ω(n) (for any heap size n). Hint: Describe an execution in which there is a delete-min operation that requires linear time and describe an execution in which there is a decrease-key operation that requires linear time. Solution a) The decrease-key operation cuts the node – together with its subtree – off the tree and inserts it as a root to the root list, decreases the key and possibly updates the pointer to the minimum. The latter is not the case here. But then we are not done yet as its parent just lost a child and has to be marked. The parent of v however is already marked, which causes it to be cut as well, together with its remaining (empty) subtree, and being inserted into the root list, again checking whether the pointer to the minimum has to be updated. This goes on recursively until we reach the root or an unmarked node (in our case the node with key 5). Note that we can unmark any root: if the root gets added to another tree, it loses the mark. If the root loses another child, we do not need to cut it off, as it is already in the root list. Thus, the resulting heap looks like this: 5 14 17 1 11 20 19 25 3 31 6 2 12 10 8 18 7 If we delete the minimum, the node with key 1 gets removed and all its subtrees are inserted into the root list (we can unmark the roots of those subtrees, if we want). 5 14 17 25 11 20 31 3 6 10 19 7 2 12 8 18 Following up is a consolidate, that merges trees of same rank. 8 merges with 19 and together they are a tree of rank 1. Which then merges with the tree rooted at 25 and is now a tree of rank 2. Like the one rooted at node v, now having the key 2. Let us merge those two trees, too. The current heap looks like this: 3 5 14 17 3 11 20 6 2 10 7 12 18 8 19 25 31 We have two trees of rank 3 and merge them, getting a tree of rank 4 rooted at v, which we then merge with the tree rooted at 5: 2 5 14 17 11 20 6 3 12 10 7 18 8 19 25 31 b) A sequence that causes a costly delete-min operation is as follows. First n elements are added to the heap, which causes them all to be roots in the root list. Deleting the minimum causes a consolidate call, which combines the remaining n − 1 elements, which need at least n − 2 merge operations, i.e., it costs Ω(n) time. A costly decrease-key operation is a bit more difficult to construct. We construct a degenerated tree. Assume we already have a tree Tn in which the root rn has two children rn−1 and cn , where cn is unmarked and rn−1 is marked and has a single child rn−2 that is also marked and has a single child rn−3 and so on, until we reach a (marked or unmarked) leaf r1 . In other words, Tn consists of a line of marked nodes, plus the root and one further unmarked child of the root. We give the root rn some key kn . We now add another 5 nodes to the heap and delete the minimum of them, causing a consolidate. In more detail let us add a node rn+1 with key kn+1 ∈ (0, kn ), one with key 0 and 3 with keys k 0 ∈ (kn+1 , kn ). When we delete the minimum, first both pairs of singletons are combined to two trees of rank 1, which are combined again to one binomial tree of rank 2, with the node rn+1 as the root and we name its childless child cn+1 (confer the picture for the current state). 4 rn+1 rn rn−1 cn ? rn−2 cn+1 ? ? r1 Since also Tn has rank 2 we now combine it with the new tree and rn+1 becomes the new root. We now decrease the key of cn to 0 as well as the keys of the two unnamed nodes and delete the minimum after each such operation, as to cause no further effect from consolidate. Decreasing the key of cn , however, will now mark its parent rn , as it is not a root anymore. Thus the remaining heap is of exactly the same shape as Tn , except that its depth did increase by one: a Tn+1 . Can we create such trees? We sure can by starting with an empty heap, adding 5 nodes, deleting one, resulting in a B2 . We cut off the lowest leaf and now have a T1 . The rest follows via induction. Obviously, a decrease-key operation on r1 will cause a cascade of Ω(n) cuts if applied to a heap consisting of such a Tn . Exercise 3: Union-Find (a) Show that when implementing a union-find data structure by using disjoint-set forests with the union-by-size heuristic, the height of each tree is at most O(log n). Hint: Show that any subtree with k nodes has height at most blog2 kc. (b) Demonstrate that the above analysis is tight by giving an example execution (of merging n elements in that data structure) that creates a tree of height Θ(log n). Can you even get a tree of height blog2 nc? Solution a) We use the hint. Note that a tree consisting of only 1 node has height 0. For k = 2 the argument also follows trivially. We prove the rest by induction. Consider two trees Tk and Tj of sizes k and j, respectively, with j ≤ k. If we combine them, Tj is added to Tk , and the new resulting tree T is of size k + j. The subtree of Tk does not contribute to h(T ), the height of T , but h(T ) is at least h(Tj ) + 1. In total we have: h(T ) = max{h(Tk ), h(Tj ) + 1}. However: h(Tk ) ≤ blog2 kc ≤ blog2 (k + j)c = blog2 |T |c h(Tj ) + 1 ≤ blog2 jc + 1 = blog2 (2j)c ≤ blog2 (k + j)c = blog2 |T |c b) For the second part, simply combine Binomial trees. Start out with n = 2k trees B0 , combine them to n/2 trees B1 . Grouping them in pairs and combining them again will give n/4 trees of type B2 . Continuing like that will result in a single Bk , which, as we know from the lecture, has height k = log2 n. 5 Exercise 4: Amortized Analysis You plan to implement a hash table. Because you don’t know how many keys will be inserted into the hash table and because the number of keys stored in the hash table might change over time, you want to be able to adapt the size of the hash table to the number of keys stored in the table. Your implementation should allow two operations Assume that you already have an efficient implementation for fixed table size s. Your implementation allows new keys to be inserted efficiently as long as the load of the hash table is less than 3/4. That is, as long as the number of elements in the table is less than 34 s, your implementation guarantees that an insert operation (which inserts one new key) costs O(1). A delete operation (which deletes one key) can always be done in O(1) time. To adjust the table size based on the number of keys stored in the table, you use the available implementation as follows. Initially, the hash table is empty. When the first element x is inserted into the table, you build an initial table of fixed size s0 and insert x. Assume that the cost for doing this is O(1). For simplicity, assume that s0 = 8. Throughout, we will always work with one instance of the available hash table implementation for a table of sufficiently large size s (which will always be divisible by 8). Operations insert and delete are implemented as follows. Assume that s is the current table size and n is the current number of keys stored in the table. insert(x): • If n < 34 s, x is inserted into the current table. Recall that this can be done in O(1) time. • If n = 43 s, we set up a new hash table of size 2s and move all items from the old table to the new larger table and insert x into the latter. We assume that the time to do this is O(s). delete(x): • If n > 18 s, x is deleted from the current table. Recall that this can be done in O(1) time. • If n = 81 s, we first delete x. If s > 8, we then set up a new hash table of size s/2 and move all items from the old table to the new smaller table. We assume that the time to do this is O(s). For simplicity, we normalize time units such that all the above operations that can be done in O(1) time need time at most 1 and the operations that take O(s) time need time at most s. Show that the amortized running times of insert and delete are O(1), a) once by using the accounting method (the “bank account method”) and b) once via the potential function method. Solution a) This method is easier to understand, but for the analysis of some algorithms like the Fibonacci Heaps, it is less useful. One easy way of analyzing amortized costs with the bank account method is to make all cheap operations as costly as allowed, but to put all excess “cash” into the account. Cheap insertions and deletions cost 1 time unit, and we are supposed to show that amortized insertions/deletions are in O(1). Thus, every time a cheap insertion or deletion happens, let us pay this 1 time unit and let us put k into our account. Let us look at a costly insertion now. The number of elements in the list is n = 3s/4 and this insertion costs s. If there never happened an increase to the table, then s = 8 and n = 6, so we have at least 6k on our account to pay for s = 8 (note that we could have much more in our account, as deletions also increase the amount of money we have). k ≥ 2 suffices. Assume now that the last costly operation was an insertion, thus the table grew from s/2 to s and now grows to 2s. In that case we had n = (3/4)(s/2) = 3s/8 elements in the list to cause the first doubling and we added at least 3s/4 − 3s/8 = 3s/8 elements in between, giving us 3sk/8 to our account. With k ≥ 3 we are 6 guaranteed to have enough “money” to pay for the costly insertion. If the last table change was a deletion, i.e., it shrunk from 2s to s and now grows again to 2s, then by the time of deletion there were 2s/8 = s/4 elements in the list. Thus we added at least 3s/4−s/4 = s/2 elements since the table shrank the last time, putting sk/2 ≥ s into our account, enough to pay for the insertion. Deletion: Similar to above we look at the history. Let a costly deletion occur from s to s/2. If it is the 2nd costly deletion in a row, the table did go from 2s to s to s/2. The first costly deletion happened when n = 2s/8 = s/4, the second when n = s/8. Thus we had at least s/4 − s/8 deletions in between, putting sk/8 on our account. We have to pay s, so we set k ≥ 8 and are fine. If the previous table change was an increase, then we went from s/2 to s and back to s/2. The increase happened when n = (3/4)(s/2) = 3s/8, the decrease when n = s/8. Thus at least s/4 deletions happened since the last table change, and with k ≥ 4 we have enough time on our account to pay for the costly deletion. b) For this one we need to find a good potential function. We defined amortized costs for operation i as ai = ti + φi − φi−1 , where ti is the actual cost. That means that we need a function φ that decreases a lot every time we have a costly operation. Remember that there is more than one potential function out there, infinitely many, as a matter of fact. A good one can usually be derived by looking at the state changes of costly operations. Let us first focus on increasing the table size. If we increase the table from s to 2s, we have an actual cost of s. Out potential function should nullify this, so ideally it is a function that has a “... − c· < table-size >” in it, because since the table size increases form s to 2s, the potential function has to decrease by at least s from step i − 1 to step i. Potential functions have to be positive, though, so we are not done yet. 1 There are not many things in the state of the hash table that we can else look at, but one does stand out: the number of elements n. For large enough k the function φi = φ(n, s) = kn − cs does the trick (except when we have an empty table, n < s/8 never happens) — but we will look at this more closely later. If we do not have any costly operation, the potential function is also not allowed to change a lot. Luckily, kn − cs suffices, as adding or deleting an element changes the potential function by k or −k, both upper bounded by a constant. However, once we come to make a costly delete from size s to s/2, the actual cost is s while the potential function increases by about s/2. Now we would prefer to have the function look like “c· < table-size > ±...”. Well, we can make a continuous function that looks like kn − cs once we get close to increasing the table size, and which looks like c0 s − k 0 n once we come close to decreasing the table size and has small value changes while we are far from any of these.  1 1 0 0   n∈ s, s  c s − k n, 8 2 φ(n, s) =  1 3   kn − cs, s, s n∈ 2 4 This function gradually decreases as n closes in from s/8 to s/2 and increases again as it approaches 3s/4. If for n = s/2 we let the first part evaluate to the second (i.e., c0 s − k 0 s/2 = ks/2 − cs or 2c0 + 2c = k 0 + k), then there is no big jump within the function as long as no table adjustments happen. 2 3 Let us be more detailed and look at a costly insert operation i when the table size before doubling is σ, and thus the number of elements before operation i is η = 3σ/4 (we use different variables to not 1 We defined potential functions to be positive in the lecture, but, as a matter of fact, they do not have to be. But one has to be careful with potential functions that can be negative, especially if it is negative after the last operation or if it could go to −∞ somewhere in the middle, thus it is advisable to create positive potential functions from the start. 2 This (small jumps) is necessary to have, as otherwise the amortized cost of a cheap operation can be very high, something we want to avoid. 3 Please consider that the choice of intervals (splitting the function at s/2) has been made at random to not overload the reader with all those things to consider and avoid while creating a potential function. As we will see later, there are much better choices for the split. 7 get confused when we evaluate φ): 3 0 3 0 ai = σ+φ(η+1, 2σ)−φ(η, σ) = σ+[c (2σ)−k (η+1)]−[kη−cσ] = σ 1 + 2c − k − k + c −k 0 (1) 4 4 0 0 We wanted k 0 + k = 2c + 2c0 , so we have that ai = σ(1 + c0 /2 − c/2) − k 0 . For that to be at most a constant, we see that we need c ≥ c0 + 2. Now we do the same for a costly deletion, i.e., η = 1σ/8: h σ i c0 0 0 0 0 ai = σ + φ(η − 1, σ/2) − φ(η, σ) = σ + c − k η − (−4) − [c σ − k η] = 4 + σ 1 − (2) 2 2 This tells us that c0 should be at least 2 and we have some final values: c0 = 2, c = 4. For k and k 0 we only know that their sum has to be 12, but we also need to make sure that the potential function is positive. This works just well for k 0 = 4 and k = 8, and thus our final function looks like this:  1 1   s, s n∈  2s − 4n, 8 2 φ(n, s) =  1 3   8n − 4s, s, s n∈ 2 4 Any operation i that does not cause any table changes keeps s the same, thus their costs are at most ai = 1 + max{k, k 0 } = 1 + max{4, 8} = 9. Note that one problem with this function was that the operation that increases the table size did compare the two different parts of φ with each other (see (1)). We could make calculations much easier if we do not make the “split” at n = s/2, but beneath 3/8 – in this case we can simply cancel out the η part of one function with the other as we did see in (2). But we can not make the split too low, as to have it also easy for the costly delete operation we need the “split” to be above 1/4. Let us try this:  1 1 0 0   c s − k n, n ∈ s, s  8 4 φ(n, s) =  1 3   kn − cs, n∈ s, s 4 4 Costly insert again: ai = σ + φ(η + 1, 2σ) − φ(η, σ) = σ + [k(η + 1) − c(2σ)] − [kη − cσ] = k + σ(1 − c) Now there is no direct connection between c and c0 anymore. We are free in our choice of c as long as it is at least 1. The calculation for a costly delete stays exactly the same. We have now our split at n = s/4 where we want both parts of the function to evaluate to the same, i.e., c0 s − k 0 s/4 = ks/4 − cs or 4c + 4c0 = k + k 0 . This gives us a simple choice for all values (note that (2) requires c0 ≥ 2): c = c0 = 2 and k = k 0 = 8. The resulting (nice) function is φ(n, s) = |2s − 8n| With this function we tried to nullify the costs that a table resizing imposes as closely as possible. However, we also can choose a potential function in which the amortized costs are much less than 0. If we increase c and c0 (then we also have to increase k and k 0 ), the amortized costs for costly operations are of the order −Ω(max{c, c0 }s), while non-costly operations have now higher amortized costs of order O(max{k, k 0 }). Now, for our final the analysis remember Pmpart ofP Pmthat the actual cost for m operations is the sum over m all ti , which is i=1 ti = i=1 ai +φ0 −φm ≤ i=1 ai +φ0 . The initial φ0 was constant (φ(0, s0 ) = 16) and all our ai were upper bounded by 1 + k = 9, leading to a constant average cost. We could have used a variety of potential functions. |2s − 8n| is probably the most beautiful one, though. 8