Start editing an editable text label, and the superscript and subscript options will appear in the "Text" format panel to the right:


Note that the text must be formatted text to get the complex formatting options. If you do not see the sub/superscript options, select the cell(s) and in the right-hand format panel, under the "Text" tab, ensure "Formatted Text" is ticked.