Let X be an algebraic variety over a field k.  Which representations of pi_1(X) arise from geometry, e.g. as monodromy representations on the cohomology of a family of varieties over X?  We study this question by analyzing the action of the Galois group of k on the fundamental group of X, where k is a finite or p-adic field.