In mathematics, the unknotting problem is the problem of algorithmically recognizing the unknot, given some representation of a knot, e.g., a knot diagram. There are several types of unknotting algorithms. A major unresolved challenge is to determine if the problem admits a polynomial time algorithm; that is, whether the problem lies in the complexity class P.
Haken's algorithm uses the theory of normal surfaces to find a disk whose boundary is the knot. Haken originally used this algorithm to show that unknotting is decidable, but did not analyze its complexity in more detail.
Hass, Lagarias, and Pippenger showed that the set of all normal surfaces may be represented by the integer points in a polyhedral cone and that a surface witnessing the unknottedness of a curve (if it exists) can always be found on one of the extreme rays of this cone. Therefore, vertex enumeration methods can be used to list all of the extreme rays and test whether any of them corresponds to a bounding disk of the knot. Hass, Lagarias, and Pippenger used this method to show that the unknottedness is in NP; later researchers such as Burton (2011a) refined their analysis, showing that this algorithm can be useful (though not polynomial time), with its complexity being a low-order singly-exponential function of the number of crossings.
The algorithm of Birman & Hirsch (1998) uses braid foliations, a somewhat different type of structure than a normal surface. However to analyze its behavior they return to normal surface theory.
Other approaches include:
The number of Reidemeister moves needed to change an unknot diagram to the standard unknot diagram is at most polynomial in the number of crossings. Therefore, a brute force search for all sequences of Reidemeister moves can detect unknottedness in exponential time.
Similarly, any two triangulations of the same knot complement may be connected by a sequence of Pachner moves of length at most doubly exponential in the number of crossings. Therefore, it is possible to determine whether a knot is the unknot by testing all sequences of Pachner moves of this length, starting from the complement of the given knot, and determining whether any of them transforms the complement into a standard triangulation of a solid torus. The time for this method would be triply exponential; however, experimental evidence suggests that this bound is very pessimistic and that many fewer Pachner moves are needed.
Any arc-presentation of an unknot can be monotonically simplified to a minimal one using elementary moves. So a brute force search among all arc-presentations of not greater complexity gives a single-exponential algorithm for the unknotting problem.
Khovanov homology detects the unknot according to a result of Kronheimer and Mrowka. The complexity of Khovanov homology at least as high as the #P-hard problem of computing the Jones polynomial, but it may be calculated in practice using an algorithm and program of Bar-Natan (2007). Bar-Natan provides no rigorous analysis of his algorithm, but heuristically estimates it to be exponential in the pathwidth of a crossing diagram, which in turn is at most proportional to the square root of the number of crossings.
Understanding the complexity of these algorithms is an active field of study.
^Grzeszczuk, R.; Huang, M.; Kauffman, L. (1997). "Physically-based stochastic simplification of mathematical knots". IEEE Transactions on Visualization and Computer Graphics. 3 (3): 262–278. doi:10.1109/2945.620492.