Let's have a back-of-the-envelope calculation:

I have a basic 2m-membered ring of n-gonal antiprisms. What are its coordinates?

The centers of each n-gon would be vertices of regular 2m-gon lying in zw plane, whose edge is equal to height of the n-gonal antiprism.

Each of them would then expand into regular n-gon of unit edge in xy plane, with alternating orientations.

The layer has 2mn vertices

The next layer (our middle one) would be made by:

1. Extruding each triangle face into a triangular prism. The edges joining two antiprisms should become squares joining two of these prisms.

2. Every lacing edge of the antiprisms will become a triangular prism. It's joined to square faces of two triangular prisms built on the triangular faces the edge joins.

3. Every vertex expands into a pair of tetrahedra. Their triangular faces will join with triangular faces of triangular prisms built on the edges.

The total number of vertices in the middle layer should be four times higher than that of the first layer, i.e. 8mn (since each vertex is joined to 4 vertices of middle layer and none of these are shared).

Reversing the operations to get next antiprism layer leads to conclusion that the second layer should be a 2n-membered ring of m-gonal antiprisms.

What we need is an algorithm, that can take the numbers m and n and compute whether the middle layer with unit edges exists, and how it looks like. Theoretically, the polychoron should close if middle layer exists for both 2m/n and 2n/m, or if it exists for 2m/m.

BTW, I took a look in Stella on sections of grand antiprism. The snub square tiling is quite clear if you know what to look for

So... how many equations are needed?

Let's assume we already have the basic antiprism ring. We know that exists.

How many types of vertices are there in the middle layer? It turns out there are just two. There's 4mn vertices A adjacent to a triangular face joining 2 tetrahedra, and 4mn vertices B adjacent to a square face joining 2 triangular prisms built on the triangular faces of the original ring.

So we need 1 equation for antiprism ring vertex - A and 1 for antiprism ring vertex - B.

In order to fix the distances in the middle layer, we need to divide snub square tiling into squares, horizontal triangles and vertical triangles. We have square/horizontal, square/vertical, horizontal/horizontal and vertical/vertical edges. I *think* that we don't need an additional equation to ensure the quadrilateral will be really square -- that should follow from the symmetry of the figure.

All in all, it looks like there are 6 equations for 8 coordinates (4 for vertex A and 4 for vertex B, one of each should be sufficient). On the first look it looks like it might work, though there are probably mistakes in it somewhere.

BTW -- for the interlocking sawtooth approach: there might still be possible to have other shapes in this category than grand antiprism: Grand antiprism has 2*5,5 and 2*5,5 ring, but other 2*m,n and 2*n,m rings might be possible as well. If m and n are not equal, these polychora wouldn't be uniform, so they might not have been found before. And my hunch is that there is connection between both types of figures... that solution for m,n in one might mean a solution for 2m,2n (or another set of numbers closely related to m,n) in the other.