For ordinary snubs, i.e. alternated omnitruncates, the new symmetry group is simply the rotational subgroup of the original symmetry group, which was generated by reflections. The new group is generated by the same reflections, but the total number of reflections applied must be an even number.
Klitzing wrote:In the example above we considered omnitruncated figures only, i.e. Dynkin symbols with all nodes being ringed, for snubbing. Now we will go on. We not only allow polytopes, the Dynkin symbols of which do use unringed nodes as well, for starting figures. We even would allow to alternate not only vertices, but even higher dimensioanl elements as well, e.g. edges, faces, cells, etc.
The procedure would be as follows. Take any Dynkin diagram of any reflectional symmetry, and decorate the nodes by any choice with ringed nodes, with un-ringed nodes, and with snub nodes. I would use as an example x3o4s, but it would apply generally.
1st step: replace any "s" node by an "x" node (here: x3o4x = sirco). This would be the starting figure.
2nd step: omit any "s" node together with all incident links (here: x3o . = triangle). This is the element to be alternated.
Alternation again takes place as above, just that we dont have pyramids any longer, but more general cupolaic elements, which are to be removed locally in an alternating way. Still there are new facets underneath the omitted elements, which come in as new facets, while the other ones are to be diminished somehow.
In our example the cut off elements are x3o || x3q (triangles atop semiregular hexagons). Thus we omit any second triangle, and replace those by those hexagons. The other triangles would be kept in place. The squares in cubical positions are to be reduced to one of their diagonals, the squares in rhombical positions are completely withdrawn together with those alternated triangles. So we would result with a figure, which is a topological variant of the truncated tetrahedron, in fact it is q3x3o.
In that example, the original symmetry group obviously is o3o4o; an element of this group is a composition of any number of reflections using the three mirrors represented by the three nodes. An element of the new symmetry group is a composition of any number of reflections using the first two mirrors (the 'x' and 'o' nodes) and an even number of reflections using the third mirror (the 's' node).
Let's call the three generating reflections f,g,h (corresponding to 'x','o','s' respectively), so we have f^2 = g^2 = h^2 = (fg)^3 = (gh)^4 = (fh)^2 = 1 (the identity transformation). The original group contains all of the following transformations; the new group contains only those where the number on the right is even:
1; (0 non-snubbed reflections, 0 snubbed reflections)
f, g; (1 non-snubbed, 0 snubbed)
h; (0 non-snubbed, 1 snubbed)
ff=1, fg, gf, gg=1; (2 non-snubbed, 0 snubbed)
fh, gh, hf=fh, hg; (1 non-snubbed, 1 snubbed)
hh=1; (0 non-snubbed, 2 snubbed)
fff=f, ffg=g, fgf, fgg=f, gff=g, gfg, ggf=f, ggg=g; (3, 0)
ffh, fgh, gfh, ggh, fhf, fhg, ghf, ghg, hff, hfg, hgf, hgg; (2, 1)
fhh, ghh, hfh, hgh, hhf, hhg; (1, 2)
hhh; (0, 3)
etc.
It seems that a general snub symmetry group is just all of those transformations where the total number of reflections in snubbed mirrors is an even number. This is clearly a subgroup; it contains the identity (as 0 is even), it's closed under composition (as a sum of even numbers is even), and it's closed under inversion (as reversing a sequence of reflections doesn't change the count of each type). It's another question, whether the subgroup is actually smaller than the original group. Note that a transformation can have many different representations, some with odd numbers and some with even numbers; consider fgfgfg=1, where the left representation has 3 f's and 3 g's, and the right has 0 f's and 0 g's.