We will explain a conjecture that expresses the Gromov-Witten invariants for elliptically fibered Calabi-Yau threefolds in terms of modular forms. In particular, there is a recursion relation which governs these modular forms. Evidence comes from the polynomial formulation of the higher genus topological string amplitudes with insertions. If time permits we also discuss the case of reduced Gromov-Witten invariants for K3 surfaces.