We examine a recent prediction for the chirality-dependence of the Casimir force in chiral metamaterials by numerical computation of the forces between the exact microstructures, rather than homogeneous approximations. We compute the exact force for a chiral bent-cross pattern, as well as forces for an idealized "omega"-particle medium in the dilute approximation and identify the effects of structural inhomogeneity (i.e. proximity forces and anisotropy). We find that these microstructure effects dominate the force for separations where chirality was predicted to have a strong influence. To get observations of chirality free from microstructure effects, one must go to large separations where the effect of chirality is at most $\sim 10^-4$ of the total force.