Emerging Visualization Solution

[benb]

@quickfur: I'd like to hear more about the UI you were describing.

[quickfur]

Well, I don't really have anything concrete to show right now, but it's based on the following observation: suppose a 4D person is standing on the ground at location P, and facing some direction F perpendicular to the vertical U (the "up" vector). If we allow F to vary while keeping it perpendicular to U, all possible F vectors together form a 2-sphere on the hyperplane of the ground that the 4D person is standing on. For any given facing direction, we may associate a unique point on this 2-sphere. Similarly, given two possible facing directions F1 and F2, we may mark two points on this 2-sphere. If the 4D person were currently facing in direction F1, and something calls his attention to look in the direction F2, the shortest way to turn his head from F1 to F2 corresponds with the geodesic on the 2-sphere between the two points corresponding, respectively, to F1 and F2. In general, this geodesic would not be parallel to any of the coordinate planes of 4-space, but would lie along some oblique direction.

Since the surface of the 2-sphere is a 2D manifold, it seems natural to map the possible directions on this manifold to the 2D degree of freedom of the mouse. Thus, one may imagine the mousepad as a flattened section (or cartographic projection of some kind) of the 2-sphere's surface, where the current position of the mouse corresponds with the point associated with F1, and some other point on the mousepad corresponds with the point associated with F2. To turn from F1 to F2, then, one simply moves the mouse from the current position to the position corresponding with F2 in a linear fashion -- this produces the geodesic between the two facing directions.

So under this scheme, the mouse would be used for pointing in the horizontal hyperplane, and some other controls would be reserved for forward/backward movement and pointing up/down along the vertical plane, presumably keyboard controls of some kind. In order for this to work well, the 4D representation needs to be symmetric in the horizontal hyperplane, since otherwise one would have to be restricted to only vertical/horizontal movements of the mouse in order not to get confusing visual changes in the display.

[benb]

@quickfur: I like it. Instead of just a mouse, one could also use a touchscreen that displays the cartographic projection of the 2-sphere's display content for additionally intuitive navigation.

What kinds of asymmetries in the horizontal hyperplane would be inimical to the proper functioning of this kind of setup?

[ICN5D]

Have you given any thought to a combination of mouse and keyboard? Like good ole first person shooter WSAD plus mouse? Or, I remember some space fighter games that used Q and E for rotation. While we're at it, how about utilizing that whole array of nine keys, QWEASDZXC plus the directional mouse? It may be intuitive, based on the extensive use of those keys for navigation. Also had a neat idea of an alternate approach to the double bowtie: how about a double circle, smaller inside larger? The larger circle is the peripheral, smoothly connecting ana/kata and left/right into a 360 deg circle. The smaller central circle would then be akin to the two center squares-view, smoothly transitioning what we see in peripheral.

[benb]

As a general rule, I suggest labeled visual examples (however crude they may be) to accompany verbal descriptions of visualization approaches. This can add to clarity as well as eliminate misunderstanding. (Hint, hint!)

[quickfur]

@quickfur: I like it. Instead of just a mouse, one could also use a touchscreen that displays the cartographic projection of the 2-sphere's display content for additionally intuitive navigation.

Not sure what you meant by display content, but obviously a 2D representation is not going to be sufficient to represent navigational information. Perhaps one approach is to show a spherical slice of the floor surface (or projection of whatever lies below the avatar), sorta like the 4D analogue of an overhead map view in 3D, with an arrow pointing in the current direction. Then as you move the mouse, the spherical slice would rotate, so you can see exactly how the floor plan rotates in response, thereby acquiring an intuitive understanding of what each mouse movement does.

Displaying merely the surface of the 2-sphere, unfortunately, isn't going to provide this kind of information; the best you can do is to label various cardinal points on the projected sphere surface as a kind of reference frame. Showing actual floor textures inside a spherical volume would give much more useful information, I think.

What kinds of asymmetries in the horizontal hyperplane would be inimical to the proper functioning of this kind of setup?

I have in mind a particular kind of reorienting rotation in 4D, in which the user rotates in-place without changing his vertical orientation and without altering his facing direction. Under the classical 4D->3D projection scheme, this rotation has the effect of rotating the contents of the 3D retina about the vertical axis, without any other visual distortions. What would be the effect with the double rainbow display? If I understand it correctly, it would show a distorting transformation of the side panels, correct? If so, then it would be rather counterintuitive for the user to turn in any direction other than the cardinal directions, since depending on the angle of the turning geodesic with the cardinal directions, the way the display changes over time would be different, thus introducing a perceived asymmetry between turning in non-cardinal directions vs. turning in cardinal directions, where such an asymmetry does not exist in actual 4D space. Thus, the user would (probably unconsciously) develop a preference for turning along cardinal directions, whereas such a preference has no real basis to a native 4D being, to whom all lateral directions are perceived equally, with no distinction between cardinal and non-cardinal directions.

[ICN5D]

As a general rule, I suggest labeled visual examples (however crude they may be) to accompany verbal descriptions of visualization approaches. This can add to clarity as well as eliminate misunderstanding. (Hint, hint!)

I totally understand. That's why I make toratope renders Okay, so, what you have so far is this:






And what I see in my mind is this:





" Centered View " should have been labeled " Intersection View of XY and ZW "


Which is an attempt to smoothly integrate both XY and ZW ortho planes into a single, perhaps more intuitive approach. As stated by quickfur, a 4D being will perceive all laterals evenly distributed in a 360 deg circle(s). So, by adapting what you already have, perhaps we can integrate and simplify those 6 squares into five distinct viewing directions. No matter what we do, trying to simulate 4D perception on a monitor won't be easy, or clean, for that matter.

The mechanics of how to apply this in the circular method should be similar to the Double Bow-Tie. Where in this case, we have condensed the middle two squares as one view, as the intersection between the XY and ZW planes. I hope I understand enough of this Hyperland build to have usable input.

The circular setup may work well with the fish-eye lensing and adjustable FOV. As for how to smooth out the transition between ana/kata and left/right, I'm not sure. Hope this helps

[benb]

@ICN5D: The only challenge that arises for me as a knee-jerk reaction about what might not work regarding the display setup you offered pertains to the content of the centered view. In the Double Rainbow setup, there are two "forward" panels which each display slightly differing content; if I recall correctly, this is because there are two axes orthogonal to the line of sight. When interacting with (or attempting to navigate toward) objects, aligning the content of both forward panels is mandatory.

One way around having two "forward" panels would be to merely superimpose the images from both onto one such that the centered view consisted of a composite image. The composite image would probably look like it was going in and out of phase to the extent that alignment between the content of the views were altered (e.g., as one rotated kataward toward an object but not leftward toward that same object). The user would probably spend a bunch of time trying to keep that center view in phase, as it were, but perhaps that is the cost of the additional degree of freedom.

@quickfur: You wrote about how "the user rotates in-place without changing his vertical orientation and without altering his facing direction." Using John's rotation inputs of UIOJKL, what input combination would yield the kind of rotation you are describing? I'm having trouble extrapolating the directions and degrees of rotation from what you described.

[quickfur]

[...]
@quickfur: You wrote about how "the user rotates in-place without changing his vertical orientation and without altering his facing direction." Using John's rotation inputs of UIOJKL, what input combination would yield the kind of rotation you are describing? I'm having trouble extrapolating the directions and degrees of rotation from what you described.

The rotation I have in mind is keyed to J and L.

[benb]

@quickfur: Fascinating. When I use those keys in Double Rainbow mode, my understanding is that my facing direction changes as per a turn to my left or right, respectively. (I use the "compass rose" geom on GitHub to help me track what/where I'm facing.)

Here is a link to it:
https://github.com/bblohowiak/Hyperland ... ity-test4b

[quickfur]

@quickfur: Fascinating. When I use those keys in Double Rainbow mode, my understanding is that my facing direction changes (by default 90 degrees) as per a turn to my left or right, respectively.

It changes your 3D facing direction, but that's merely an artifact of the second projection from the 3D retina to the 2D screen. Note that all of the rotation keys you listed does not change the 4D facing direction at all. They just change your orientation in 4D. Of these, two pairs of rotation keys also change your vertical orientation; but J and L preserve both vertical and facing directions. The actual change in facing direction comes from the (lowercase) j, l, u, o, i, k keys.

Or is that where the confusion comes from? I'm referring to shift-J and shift-L, not (lowercase?) j and l.

[benb]

@quickfur: My apologies; you are correct, I interpreted your J and L as j and l.

"I have in mind a particular kind of reorienting rotation in 4D, in which the user rotates in-place without changing his vertical orientation and without altering his facing direction. Under the classical 4D->3D projection scheme, this rotation has the effect of rotating the contents of the 3D retina about the vertical axis, without any other visual distortions. What would be the effect with the double rainbow display?"

With Double Rainbow, indicators of facing direction and vertical orientation remain unchanged. Depending on direction of rotation, indicators of left/right swap places with indicators of ana/kata.

There is no distorting transformation of the side panels if the Flare mode is deactivated; the cubes appear as square as the center indicator tesseract does. Granted, the cubes grow and recede a bit through the rotation, but they do not deform per se. Even with the distorting transformation of the side panels in Flare mode, the visualization of the undistorted tesseract appears as such when seen on the center panel.

As for the display predisposing individuals to turning in cardinal directions, I very much enjoy and have logged plenty of time turning in increments of fewer and greater than ninety degrees. That said, I do believe the biggest challenge you're alluding to is in getting the "two centers" of the forward views aligned on target objects, destinations, etc. With attentive manual control, one can pilot through a 4D obstacle course with a minimum of collisions. At the same time, I believe that there exist improvements to be made above the Double Rainbow Flare design as it presently exists.

[ICN5D]

The only challenge that arises for me as a knee-jerk reaction about what might not work regarding the display setup you offered pertains to the content of the centered view.

Well, I was thinking about that, and you're right. They're uniquely different, for good reason. I'm not sure yet how to combine the two distinct directions into one. I'll have to play with the actual program and get a better feel for it. I don't think superimposing two views would work ... too confusing. Hmm. But, what do you think about the placement of ana/kata? Rather than this 4D up/down set up in a right/left direction, it would be simply up/down, from the intersecting view. And, couldn't we rotate what's left/right to be ana/kata? If so, this transition could happen through nothing more than sliding everything around the peripheral circle. Something to consider ......

[benb]

@ICN5D: I wouldn't dismiss the possibility of superimposing the views out of hand as too confusing. I think that some testing might be in order, especially given the experiences I've had with manually aligning the two views when they are separate--it can be tough, but it can be done. One option that occurs to me regarding how this might be accomplished in addition to the manual incremental rotation/orientation control is through some kind of system analogous to compass navigation; rather than visually seeking the same indicator of the same forward direction in both forward views, perhaps a user could aim both forward views toward a common point in the x,y,z,w manifold via coordinate entry. To the extent that the images were misaligned or "out of phase"-looking, rotations and manual adjustments could compensate.

And, as for rotating left/right and ana/kata, via the circular display you charted I'm open to that. My basic predisposition is to test it out to get a feel for its affordances and limitations. Can you write the code to make it happen, or should we look for ways to make it so?

[ICN5D]

Well, I did program the heck out of my TI-86, but that was a looong time ago. Seeking a way is the only way, I guess. Maybe quickfur knows a way?


I'm still thinking about how to represent the forward view. But, first off, what are the flattened blue lines on the "ground" in each view panel?

[quickfur]

I use the "classical" 4D -> 3D projection method for all 4D visualization, because I find it easiest to understand, reason about, and to develop geometric intuition with. It has served me surprisingly well, considering that it does have well-known limitations (I rediscovered many 4D shapes by direct visual construction using projections, and my discovery of the castellated rhombicosidodecahedral prism (D4.3.1), the first known non-trivial polytope with bilunabirotunda cells, was also done primarily via construction with projections).

But Ben here prefers alternative approaches, and since this is his thread, I'll refrain from pushing my angle.

[benb]

@ICN5D: The blue grid is a "mat" that the objects sit on. It can help with reference/orientation (i.e., which way is down). What are your thoughts on it?

@quickfur: I'm here to receive input as much as I am to share output; the "classical" projection method (or anything else) should be on the table as long as it's not violating community norms by being mentioned. Hyperland is an open source project and the Double Rainbow versions are among its emerging fruits. If the "classical" projection method can help make progress toward accomplishing project aims (e.g., manipulating photorealistic representations of objects in a simulated physical space that consists of four dimensions, users experiencing moving themselves and/or non-self objects toward intended target destinations in that space, etc.), then I am for inclusion of that method, as it were.


Because of the open nature of the Hyperland project, those who participate in this thread and would like official credit for their contributions (rather than just a special thanks), please let me know in a private message.

[quickfur]

Photorealism is something that I've been long dreaming about when it comes to 4D visualization. My current polytope viewer's approach, which is rendering ridges (i.e. filled polygons) with transparency, is only a first approximation of what I envision. It is primarily limited to objects with flat surfaces, i.e., polytopes, and perhaps a small class of simple curved objects. But even something as simple as a 4D sphere is difficult to adequately convey -- because the true curvature of the 4D sphere can only be represented by a 3D gradation of shading across the projected spherical volume! For this, it seems that 3D volumetric rendering is a necessity. Unfortunately, how to convey such a thing adequately in the limited medium of 2D screens is a major challenge. The next logical step would be 3D volumetric displays -- but I neither have the means nor the expertise to be able to construct such a device!

[benb]

What about binocular output? That seems rather straightforward.

[quickfur]

My website already uses stereographic viewing, for example:



(This is a cross-eyed stereo pair: to see the 3D image, cross your eyes so that the two images blur into 4, and adjust the amount of crossing until the middle two merge. Then focus on the merged image and ignore the other two, until the middle image comes into focus.)

This particular example is one of the better ones I've done, in that most of the important elements are clearly perceived in the 3D result. However, it still suffers from an inherent limitation: there is only so much you can cram into a 2D representation of a 3D projection image before things become too cluttered. Even in this example, it's rather difficult to see the faces on the far side of the yellow great rhombicuboctahedron -- yet, from a 4D point of view, this yellow great rhombicuboctahedron ought to be in the center of the field of vision, so its elements should be the clearest of all.

Now, from a 3D-centric point of view, the above image only has a depth of about 4-5 polygons at the most (i.e., only about 4-5 polygons at the most lie along the 3D line of sight), and already, the cluttering problem is significant. Imagine now if, instead of merely 4-5 polygons along the line of sight, you need to represent a dense array of pixels representing the projection of a curved surface, say the output of a 4D raytracer with smooth light and shade. It quickly becomes an amorphous blob for anything but the simplest of shapes.

Even if we stick with polygonal rendering (no curved shapes like 4D spheres), 4-5 polygons is already pushing the limit of visibility. In a complex 4D scene with multiple faceted objects, 4-5 polygons per line of sight is far too few. Some other way of getting around the practical limitations of the 2D screen would have to be adopted. One thought I have along these lines is, if you imagine the classical "cube-within-a-cube" projection of the tesseract, the volume in the "front" frustum (i.e., closest to the 3D point of view) can be elided or rendered with higher than normal transparency, so that the volume in the inner cube (i.e., where the closest point of a convex object to the 4D point of view would be projected) would be less obstructed to the 3D viewer. There would then be auxiliary controls where the user can easily spin the 3D viewpoint (equivalent to the J/L 4D rotation that we talked about earlier, but not necessarily bound to an actual 4D motion -- I like to think of manipulations of the 3D viewpoint as separate from actual 4D movement, being merely crutches to help us poor 3D creatures grapple with 4D), thus rotating the "front" frustum away from the elided/high-transparency region when they wish to see more clearly the part of the projection image that lies in it.

Alternatively, there could be a "sphere of focus" associated with the 3D view, where a dedicated set of controls move a virtual sphere within the 3D retina, and the portion of the projected 4D scene that lies within this virtual sphere will be rendered more solidly than the rest of the contents of the 3D retina, thus providing the user a way to explore the 3D retina without needing to compromise on the visibility of various elements projected within it. For real-time interaction, full 3D freedom of movement for this virtual sphere would be too taxing to control, so one idea is to default it to the center of the 3D retina where most of the elements of interest would fall on most of the time, and to have 6 keys that will translate the sphere to one of the 6 regions corresponding with the 6 frustums of the tesseract projection (i.e., up/down, left/right, front/back) temporarily. Sort of like "casting a glance" in one of the 6 directions, where upon releasing the key, the sphere of focus returns to the center of the retina.

Yet another idea is simulated 4D camera defocusing, where the object that lies directly in front of the 4D viewer will be rendered with sharp edges, but the farther out from the line of sight an element is, the more blurry it is rendered (and the higher the transparency), so that objects on the periphery of the 4D view will just be a defocused amorphous blob, and the important part of the projection, the central volume, will be most clearly visible.

All of these ideas would be good to explore, but unfortunately they all require significant implementational effort and graphics programming expertise, and I don't really have the time/energy to be able to do all of this.

[quickfur]

Also, another line of thought w.r.t. photorealistic rendering of 4D objects, is that what we 3D being perceive as solid 2D surfaces, are no more than margins or ridges (i.e., the equivalent of edges) to a 4D viewer. So any "photorealism" like my use of povray to render these polygonal projection images are nothing more than wire diagrams with fancy edge colors, from the perspective of a 4D viewer -- a far cry from 4D photorealism!!

Indeed, all of the ideas I described in the above post are ultimately an effort to grapple with the fact that 4D objects have 3D boundaries, and how to adequately convey the full 3D-ness of these hypersurfaces in a way that's comprehensible to us poor, 4D-challenged 3D folk. While it is not too difficult to describe a 3D manifold in the way of delineating its boundary, 4D photorealism requires realistic light and shade along every point along the 3D volume of the manifold, something which our 3D-centric biological sight has no capacity to handle. After all, our eyes only ever see 2D; our perception of 3D depth is entirely a construct of our mind. How do you convey a complete contents of a 3D manifold (a 3D array of pixels) to the brain when it has no such channels that can carry such information?

Perhaps asking for 4D photorealism is a bit too ambitious -- that's like a 2D being trying to understand a photograph of complex 3D scenery when their poor 2D eyesight can only see an infinitely thin slice of the photo at a time. The only true way to surmount this inherent barrier is if we can somehow transmit a 3D array of visual signals to our brain directly, and if our brain can somehow adapt to this augmented stimulus and understand it as visual input. But even if this is possible at all, it is definitely far from the current reach of technology. So in the meantime, the best we can do is to see how far we can get with our limited 2D retinas.

[benb]

@quickfur: You ask:
"How do you convey a complete contents of a 3D manifold (a 3D array of pixels) to the brain when it has no such channels that can carry such information?"

Your answer was in your text: "...our perception of 3D depth is entirely a construct of our mind."

Thus, I would suggest that our perception of 4D may as well be a construct of our mind, as well. While binocular depth cuing seems compatible with our commonplace 3D depth perception, I would anticipate that utilizing both input streams (with differing content) would be optimum for 4D representation.

I would prefer a setup that delivered content directly to each of my eyes rather than crossing them. As for how that content is organized, your suggestions make much sense.

I may take exception to this one, though:
"While it is not too difficult to describe a 3D manifold in the way of delineating its boundary, 4D photorealism requires realistic light and shade along every point along the 3D volume of the manifold, something which our 3D-centric biological sight has no capacity to handle."

Perhaps this speaks to assumptions we've been making about interiors and exteriors, at least as far as principles of sight or light reflection are concerned. In the spirit of questioning assumptions, what prevents us from visualizing every point of an object?

[quickfur]

@quickfur: You ask:
"How do you convey a complete contents of a 3D manifold (a 3D array of pixels) to the brain when it has no such channels that can carry such information?"

Your answer was in your text: "...our perception of 3D depth is entirely a construct of our mind."

The problem is that the 3D construct in our mind is based on 2D surfaces. It can only represent a 2D subset (possibly curved in a complex way, but nevertheless merely 2D) of the 3D array of pixels.

Thus, I would suggest that our perception of 4D may as well be a construct of our mind, as well. While binocular depth cuing seems compatible with our commonplace 3D depth perception, I would anticipate that utilizing both input streams (with differing content) would be optimum for 4D representation.

Believe me, I have tried doing 4D binocular rendering for either eye. It doesn't work. Our brain simply isn't wired to interpret the images in a 4D way, at least as far as input from the eyes are concerned. Our visual apparatus seems to be hard-coded for 3D perception (unsurprisingly!).

I would prefer a setup that delivered content directly to each of my eyes rather than crossing them. As for how that content is organized, your suggestions make much sense.

I'm not sure what you mean by "rather than crossing them". The whole point of a cross-eyed stereogram is to allow arbitrary width of each of image pair. It's trivial to make a wall-eyed stereogram (just exchange the images in the pair), but since our eyes generally can't diverge, this limits the maximum width of the images. The cross-eyed approach allows, in theory, arbitrary width because our eye muscles can cross the eyes, but they can't push them apart. But in either case, the overall result is the delivery of two disparate images to either eye in order to achieve stereopsis.

I may take exception to this one, though:
"While it is not too difficult to describe a 3D manifold in the way of delineating its boundary, 4D photorealism requires realistic light and shade along every point along the 3D volume of the manifold, something which our 3D-centric biological sight has no capacity to handle."

Perhaps this speaks to assumptions we've been making about interiors and exteriors, at least as far as principles of sight or light reflection are concerned. In the spirit of questioning assumptions, what prevents us from visualizing every point of an object?

Nothing prevents our brain from visualizing such a thing, obviously. The problem lies in how to deliver such information to the brain in the first place! I didn't find any specific numbers as to the resolution of light-sensitive cells in the human retina, but it's pretty high-resolution, say about 10,000 pixels. So in theory, our eyes can see a 2D 10,000*10,000 array of pixels individually (I know this is not 100% accurate, because cell density varies across the retina, but let's for now postulate this simplifying assumption). Now imagine how many different curves can be represented by a 2D array of 10,000*10,000 pixels. Conceivably, one could discern complex curves up to a density of 5000 distinct lines across any single line drawn across the array.

Now suppose we have an analogous 3D array of pixels, which is 10,000*10,000*10,000. Imagine the maximum complexity of a 2D manifold that can be represented in such an array. Conceivably, it can curve in a complicated way up to a density of about 5000 distinct surfaces across any single line drawn across the 3D array. OK.

The challenge now is, how to convey this complex 2-manifold to our brain using only 2D images. The most obvious way is to display 10,000 slices of the 3D array, one at a time... this transmits the total information, however, it loses the connectivity of the 2-manifold across the 3D volume of the 3D array. We see a slice at a time, and we can see the complicated 2D cross-sections of the manifold, but only those that are parallel to the direction of the slices. You may not necessarily be able to put the slices together mentally in order to form a continuous 3D model of how the complex 2-manifold curves itself in the 3D volume of the array. In fact, even a relatively simple shape that forms simple cross-sections presents difficulty for our brain to synthesize a continuous 3D model, for example:



Each yellow shape represents a cross section of a particular 3D shape. Here there are only 5 cross sections, but the shape is simple enough it's easy to interpolate between them. Can you tell what the 3D shape is? If you've seen this before, probably yes. If not, it's unlikely, or at least, requires much mental effort before a coherent 3D model is conceived.

Now imagine a 3D retina whose contents are far more complex than such a simple 3D shape, and now we're not talking about a handful of cross-sections, each of which is a simple shape easily interpolated between each other, but a complex curve with potentially completely different cross-sections at almost every point. It stretches credulity to believe that our brain will easily be able to reconstruct the full 3D contents of the 3D array of pixels! In any case, real-time visualization via this method is out of the question.

OK. What other methods do we have besides slices? Perhaps we can use stereopsis to convey the shape in its native 3D space by presenting a stereo pair? Assuming a complex-enough 2-manifold drawn in the 3D array of pixels, it may have thousands of mutually connected layers that fold back upon themselves, branching and merging in many different 3D orientations. We have already established that after about 4-5 surfaces that lie along a single line of sight, the 2D projection images are already so cluttered that it's difficult to see every part of the 3D shape thus presented. But here we're talking about thousands of layers of a single 2-manifold that curve and warp in a very convoluted way in the 3D volume of the 3D pixel array. Again, it's infeasible to represent the entire complexity of such a thing in a single image, even if it's a stereo pair. We would have to present it piecemeal, which again leads to the situation that, given 1000 such piecemeal presentations of the 2-manifold (1000 by the assumption that we can maximize the transmitted information by displaying up to 5 distinct surfaces along a single line-of-sight each time, given a curve complexity of 5000 parallel surfaces at maximum density), can our brain easily form a complete mental model of the entire thing? It should be much easier than the cross-section approach, for sure, but you'd still have to look at 1000 stereo pairs before you have enough information to reconstruct the whole thing. Real-time visualization? Not a chance.

Now all of the foregoing concerns only a monochromatic 2D manifold drawn into the 3D array of pixels. 4D photorealism isn't merely a single monochromatic 2D manifold; we're talking about an arbitrarily complex 3D pattern of colors across the pixel array! There may or may not be a pattern of surfaces you can decompose the image into, that you can employ stereograms on. If the pixels represent a complex, dense texture of a coarse, grainy 4D surface, for example, the only way to fully convey the entire texture would be the cross-section method, since potentially every slice of the array will have almost no continuity with the previous slice! Sure, if our brain can somehow receive all of this information in one go, then in theory we could perceive the entirety of the contents of this 3D array at once. But given that currently the best channel of transmission is via our 3D-challenged eyes, surely you must agree that this stretches credulity past its breaking point! Visualization in real-time, therefore, is completely out of the question.

So that brings us back to the inherent limitation that we can only feasibly visualize extremely simple 4D scenes, that consist only of relatively simple objects in relatively simple arrangements. While it may be theoretically possible to go somewhat beyond that, it seems to be a stretch to suggest that we can grasp an arbitrarily complex scene, much less do so in real-time! Consider, for example, a typical photorealistic 3D scene. It requires pretty much every pixel in the 2D projection image of the scene. Now imagine how a hypothetical 2D creature might perceive such an image. From its disadvantaged POV confined to the 2D plane, it cannot see the image in its entirety, but can only perceive 1D slices of it at a time. Equivalently, this is like trying to play a 3D first-person shooter with a 1-pixel wide screen. How well can you visualize the total scenery under such a deficient vision? Probably not very well, if at all!

The same challenge applies in going to 4D scenery with our deficient 2D vision.

[ICN5D]

I've been thinking of this 4D vision.... ultimately it is a 3D retinal scan. This can be represented by a whole 3D world to explore, but with a built-in extra depth to everything. One could roam around the scan to see its entirety, and explore the extra 4D parts at will. We would have a way to perceive 4D depth, in the form of transparent refractive extensions branching off of all 3D structures and land forms in the retinal scan. These extra 4D parts look like ultra-clear glass. So clear in fact, that you wouldn't actually see the glass, but the bending of the light caused by it. Everything in the retinal scan would have this non-existent, yet still perceivable extra parts. The invisible parts would be non-interactive, so you could walk through them ( advantageously). Some objects would seemingly appear out of nowhere, once one rotates or translates to it. Maybe a few sub-windows to represent two orthogonal top-down views would be useful. These views would be 3D environments that show in 3rd person where the full structures are filled in.

The main focus would be the centralized view of a 3D retinal scan, with the addition of orthogonal top-down views. This would act like a compound eye, seeing in multiple directions at once. Moving along the new linear 4D extension would fill in parts of the transparent extensions, while causing filled in parts to disappear. Say for instance, you reached an impassable wall on a journey. Then, you noticed there was something else there, hidden in plain sight. So, you rotate or translate in 4D and transform the barrier into a tunnel. Essentially you would perceive a 3D environment, where it can be manipulated and altered at will by motions in 4D. You would use this ability to navigate obstacles, being forced to think in 4D. Judging how the solid and transparent parts are assembled, you would have to navigate four dimensionally and move yourself, including your 3D retinal scan, around to find the right path.

I also watched that 4D maze youtube video, it was really cool! The rotations are very interesting, and I see why you use six squares for vision. I checked out the websites, but couldn't figure out how to run the program. What exactly do I need to do?

[benb]

@ICN5D: To run the program, you need to use Terminal. Navigate in Terminal to the directory named "out." Then, type "java Maze" and hit "enter."
That should get you started.

[benb]

@quickfur: "Nothing prevents our brain from visualizing such a thing, obviously. The problem lies in how to deliver such information to the brain in the first place!"

I agree! Perhaps the meaning of the word "photorealism" in this context must be reconsidered and clarified. How, exactly, do we define the successful photorealistic representation of an object in fourspace?

My knee-jerk reaction is to say that success is when an object looks like (and could be mistaken for) something real, with a great degree of nuance and texture and light and shadow as part of its appearance in its environment. I am aware, however, that there are more sophisticated considerations at stake. I am curious as to what the community thinks about photorealism in this sense; perhaps it should be its own thread. Thoughts?

[quickfur]

My working definition of "photorealistic" would be the direct analogue of 3D raytracing: in 3D raytracing, you have a digital model of some 3D scene containing various objects, and the raytracer produces a 2D array of colored pixels based on tracing lines of sight from some specified camera position + orientation. It has already been proven that raytraced images can exhibit a high degree of photorealism.

So I'd say 4D photorealism, in the most direct sense of the word, would be the result of 4D raytracing, where the raytraced image is represented by a 3D array of pixels created by tracing lines of sight in 4D from some specified camera position + orientation.

How such an image can be presented to the user in its full glory, though, remains an unsolved problem. Direct representation seems impractical beyond the most trivial cases, since raytraced 4D objects would presumably have 3D textures applied to their surfaces, complete with light and shade, and these textures can be arbitrarily complex. Thus the result cannot be adequately represented by any 2D method (except perhaps by slicing, which precludes real-time interaction).

In any case, 4D raytracing isn't a new idea by any means. One Steve Hollasch (search on google) has created a working prototype using the slicing method to render the final result, and this was many years ago. However, slicing comes with all the disadvantages associated with it: it is difficult to get an adequate feel for the 3D-ness of the raytraced image, and equally difficult to synthesize that into something that one can then extrapolate back into 4D.

One thought that occurs to me, though, would be to perform visual recognition on the 3D image in order to extract features of interest from it, using image recognition techniques (suitably generalized to 3D pixel arrays). These features of interest, which would include things like the location and shape of sharp boundaries, highlights, etc., can then be translated into simplified representations such as, perhaps, 2D manifolds (contour surfaces?) embedded in a 3D model space, which would be significantly simpler than the original full 3D textural information. These then can be rendered directly on the screen using 3D rendition techniques, with their 3D location conveyed using stereographic techniques (red/blue glasses, cross-eyed/parallel stereograms, 3D TV, etc.), with suitable application of (possibly selective) transparency to alleviate the occlusion problem. But then this reduces essentially back to our current method of rendering only 2D surfaces, so it fails the definition of photorealistic. (And, sadly, this exemplifies the inherent difficulty of conveying full 3D information via a 2D-only channel.)

Nevertheless, image recognition techniques would be able to pick up things like prominent light/shade boundaries that are not present in the geometric representations of the target objects, so it could still give a faint glimpse of "true" 4D photorealism.

[benb]

Interesting. Let's keep going.

If we assume a fourspace with roughly equivalent physics (light behaves with wave/particle duality and in other Newtonian and quantum ways) to that of threespace, before a 3D retina can register the results of electromagnetic waves they may first be focused by a 4D lens. How much consideration has been paid to 4D lenses in this context--how do they behave and what affordances may they provide?

[quickfur]

Hmm. I haven't really thought much about 4D lenses... though I presume they will be approximately in the shape of a flattened 3-sphere, so they will exhibit analogous properties to 3D lenses.

However, if we are to take into account physics, then the situation is drastically different from the usual idealized dimensional analogy from 3D. There are a number of currently-known drastic differences:

- Quite early on, a member of this forum (Patrick Stein aka pat) solved the equations of orbital motion in 4D (under the assumption that gravity diminishes according to an inverse cube law, in accordance with the flux theory of force propagation), and discovered, to the chagrin of all of us, that 4D orbits are inherently unstable. Elliptical orbits simply do not exist. The only stable orbit is a perfect circle -- and the slightest deviation from the mathematically perfect circular curve means devolution into an unstable spiralling path that degrades linearly over time -- clearly an untenable proposition for stable orbital systems! Large deviations from the perfect circle result in rapid collision with the central star or speedy escape into outer space.

- Later on, we found a paper that proves that the Schrödinger equation for the hydrogen atom (i.e., 1 proton + 1 electron) in 4D has no non-trivial minima. Meaning, atoms as we know them do not exist in 4D -- a proton/electron pair will simply collapse. No discrete electron energy levels / orbitals are possible, hence chemistry is impossible, and physical materials as we know them cannot exist. If physical 4D objects are to exist at all, they must be of a fundamentally different nature than what we know from our own universe.

- Then recently, another forum member found a reference that indicated the unusual behaviour of waves in spaces of even dimension. Upon investigating this claim, I found a paper that showed that signals transmitted via waveforms can only be received in pristine form in 3D. In all even dimensions (including 2D), waves propagating from a point source exhibit an "echo effect", where the advancing wavefront repeatedly sends back "echoes" that interfere with the trailing wavetrain and cause distortions of the signal being transmitted. Thus, if a single pulse were sent from the point source, a receiver located some distance away will receive not a single pulse, but multiple (distorted) copies of the pulse, echoing in complex ways and gradually fading as the initial energy dissipates. The 2D case of this effect can be seen when you drop a stone into a still pond: the circular wavefront moves outward, but also inward, interfering with itself, reconverging at the source and re-emerging as secondary, tertiary, quaternary, etc., wave fronts, each of which sends back more echoes, which in turn produce more re-emerging fronts, which send back more echoes, etc.. In odd dimensions above 3, the echo effect is absent, however, in 5D the signal received by the receiver is not the undistorted original signal, but its derivative. In 7D, the received signal is the second derivative, and so on. 3D is the only dimension where propagating wavefronts can transmit a signal without any inherent distortion.

Taken together, all of the above indicate that if we are to generalize 3D physics to 4D, it would result in a universe so foreign that we would barely even be able to begin to comprehend it, much less recognize anything that might even remotely resemble life as we know it. In particular, the lack of stable orbits means that galaxies, planetary systems, moons, etc., cannot exist in 4D (at least, not long enough to sustain whatever analog of life we may postulate!). The lack of stable atoms means that either we postulate an entirely different kind of atom than what we understand in our own universe, or physical objects in 4D must be of a radically different nature. Lastly, the echo effect of waves means that light will behave in a fundamentally different way than we expect.

Assuming that light arises through as the force carriers of whatever analogue of electromagnetism we postulate (which in itself is highly problematic because the cross product in 4D is a bivector, not a vector, so it is unclear how Maxwell's equations would be generalized at all), its wavelike nature would cause it to interfere with itself in an echo-ey way, so a single flash of light would appear to an observer as a blinking light flashing multiple times in succession. The inability of 4D waves to carry a pristine signal means that color (assuming it arises from wavelengths of light) will warp and shift over time due to the self-interference of light under the echo effect, probably resulting kaleidoscopic visual distortions that bring the very nature of vision itself under question.

TL;DR, a truly native 4D world would be extremely alien and quite incomprehensible, and probably defeats the purpose of 4D visualization in the first place, because these secondary effects of the extra dimension would take too much prominence, thus obscuring the geometric aspects of it which (I presume!) is the whole point of the exercise. Therefore, I'm perfectly happy to live with "naïve" dimensional analogies from 3D that don't necessarily have to make sense w.r.t. a "native" 4D physics -- because that's what gives us the tools to develop an intuitive feel for 4D space. While it's an equally (if not more) interesting challenge to study a truly native 4D universe, it also requires far too much more effort than is worthwhile for what amounts to nothing more than a fantastical world that, on top of being completely fictitious, is incomprehensible to any potential audience.

[benb]

@quickfur: Thank you for the thorough reply, and for citing specific examples on this board. I think that it is important to address some of these issues when staking out "territory" as far as photorealism is concerned. Both the reality and the photographic processes under consideration must be explicated.

"The inability of 4D waves to carry a pristine signal means that color (assuming it arises from wavelengths of light) will warp and shift over time due to the self-interference of light under the echo effect, probably resulting kaleidoscopic visual distortions that bring the very nature of vision itself under question."

I find this to be noteworthy and perhaps something worth pursuing.

The impossibility of orbital motions and atoms doesn't bother me much--that's what other dimensional realities are for.

As may be evident, I have a longstanding interest in how fluids may behave (and be modeled) in fourspace. I would include models of electromagnetism as a fluid within that broader umbrella.

"a truly native 4D world would be extremely alien and quite incomprehensible, and probably defeats the purpose of 4D visualization in the first place, because these secondary effects of the extra dimension would take too much prominence, thus obscuring the geometric aspects of it which (I presume!) is the whole point of the exercise."

I'm not sure to what extent it is wise to separate the polychora from the other properties of the manifold in which they may arise. For all we know, we may be stipulating the existence of figures that abide by certain rational principles while giving short shrift to other pertinent principles; of course, any physicist might make that claim to any geometer in an effort to bring the two efforts closer together. The difference in this case is that we do not have empirical evidence from fourspace to inform our hypotheses regarding how objects possess mass and exert force upon each other--but we can tweak the parameters that would account for "secondary effects" in ways that may be more or less conducive to the "natural" arising of polychora--reverse engineering a cosmos in which polychora could assemble from constituent elements/forces...