Definition
Meta-mathematics is the study of mathematics itself as a formal object. Where ordinary mathematics proves things about numbers, sets, spaces, and functions, meta-mathematics proves things about proofs — about which statements are provable in a given formal system, what such systems can and cannot express, whether they are consistent, and whether they are complete.
David Hilbert coined the term Metamathematik in the 1920s as part of his program to settle the foundations of mathematics by formalizing every mathematical practice and then proving, within a safer fragment of mathematics, that the formalization is consistent. Gödel's incompleteness theorems were the main results that showed Hilbert's program could not be carried out as originally hoped.
Why it matters
How it works
To do meta-mathematics, you first specify a formal system as a precise object — its alphabet, well-formed formulas, axioms, and inference rules. Then you reason about the system using ordinary mathematical methods. You can ask: which formulas are theorems? Is the system consistent (does it never prove both A and ~A)? Is it complete (does it prove every truth in its language)? Is it decidable (is there an algorithm that, for any formula, decides whether it is a theorem)?
Gödel's trick turned meta-mathematics back on itself. He encoded each formula and each proof as a number, so that meta-mathematical statements ("formula f is provable") became arithmetic statements about those numbers. Arithmetic-rich formal systems can then express meta-mathematical claims about themselves — and that self-application is what produces incompleteness.