Not necessarily – identical twins don’t necessarily create identical babies, there’s a bit more random selection in there in terms of which genes get inherited. However there would probably be similarities in their appearance just like there usually is for most brothers, sisters, cousins etc.
As you probably know, each twin in a pair of identical twins inherits two alleles of each gene: one from each parent. Both twins will have the same pairs of alleles as each other for every gene. However, when one twin from set A reproduces with one twin from set B, either of the alleles for each gene will be passed onto the child. When the other twin from set A reproduces with the other twin from set B, it won’t necessarily be the same alleles that get passed down from each parent. So they will only be as alike as siblings from the same parents would be.