Augmented Reality Snorkeling

Project Status: Still Dreaming

The Idea

I would love to design a mask for divers/snorkelers with an integrated heads-up display, commonly referred to as a HUD. I would like to have a clear window for the swimmers to see what is going on around them as they normally would and I would want to project or reflect images onto this window.

The device would process what is visible to the diver and augment this field of view with interesting information. This is a hands-free device, though the user could control the system by making hand gestures in front of the window. For example, acting as if you were wiping the screen could clear the interface.

I would take advantage of both speech processing and audio communications from other divers nearby. Thus allowing more control over the interface and more interactive fun with other people wearing the same headsets.

A Starting Point

Check out the H2O Ninja Mask featured in the video below. Here you see a product that could easily be altered to develop this underwater HUD gaming/educational platform.

Similar Projects

After a quick search I did come across something called Project Nautilus VR which had a Kickstarter page. Sadly it looks as though the project did not receive enough backing to get going. You can check out their video below.

Project Nautilus VR is an attempt at virtual reality, meaning they replace and control your entire view. I would rather a project that is augmented reality, meaning you still see the outside world but you add content to it. Although you could easily cheat an augmented view with their project by showing the live camera and drawing on top of it, my main concern is safety when the system lags or is buggy. I also believe the experience you have and the realistic feeling are both diminished when you jump from augmentation to a complete virtual view.

I'm not sure what happened to these guys or what they are up to today if they aren't still working on Project Nautilus VR, but whatever it is I wish them the best.

Product Uses and Real World Applications

Not sure why you would want an underwater heads-up display with augmented reality? Whether you are in the backyard pool or the Caribbean, honestly the possible uses are only limited by your imagination. Let me give you a few ideas:

• Education:
• Want to know what kind of fish that is in front of you?
• How about finding out some more information about that coral reef you shouldn't be touching there?
• What if you could project different species to learn about onto your mask?
• Health / Fitness:
• Want to race against your best time from previous swims?
• Would you like to track and learn from your swimming stats, in real-time?
• Gaming:
• How about some capture the flag, maybe with virtual bases and communications between team members?
• Want to see if you can play keep away, with some virtual sharks?

Machine Learning to Improve 360 Video Playback

Project Status: Still Dreaming

Problem

Watching a 360 video in any setting other than virtual reality could lead to a poor experience due to non-optimal viewing path. Viewers have to work pretty hard to be their own directors of photography or else they may end up staring at the wall while something super awesome is going on in the other direction.

Workaround

Currently you have to back up and try to find what you missed, assuming you even realize that you missed something in the first place. Using any type of virtual reality headset will help immensely due to the simple fact that the viewer may naturally look around in the environment.  

Solution

Storing viewer paths as people watch 360 degree videos will give us the data needed to learn what path is optimal to suggest to a viewer who has never seen this video before.

So while you won't be physically pushing a person's head around as they wear a virtual reality headset, you can be pushing the view around for someone watching from an online source such as YouTube. Meanwhile for those VR headset users, we can still suggest directions they may want to look in much the same way video games do. If you are as horrible of a gamer as I am you will know right away that flashing red area on the side of the view means that is the direction you are being attacked from.

When a new 360 video is uploaded to a site such as YouTube, one cool option would be to give the person uploading a tool that allows him/her to set a default viewing path throughout the video. Honestly that should be done anyway but, as you know, we can do even better that.

Now to be clear, what I mean when I say a path is basically if you imagine recording not a whole copy of the 360 degree video and every direction but the actual video frame by frame that a viewer saw. While watching the viewer may have the ability to turn in any direction but ultimately they can only look one direction at a time and so whatever they choose to see becomes a video experience of its own. Even if they back up and play through parts again while checking out other directions their perspective is still limited to one direction of view at a time.

So obviously it wouldn't be very efficient to store a new copy of the video each time it is viewed. Instead what we actually need to store is the camera angle at each point in time. We store this frame by frame camera angle and call it a path.

For now we'll keep the idea simple with camera angle data but it is interesting to note that very soon with the advantages and expanding use of virtual reality technology tracking eye movement will also be extremely useful to use.

Maybe other features could help us learn quicker such as if we take into account when the user backed up to view something else. We could also note when the viewer is using a virtual reality headset, because when they are we will end up weighing the path importance differently due to the fact that they have easy means of motion. Some paths may be more traveled depending on screen sizes or viewer demographics. We should focus on anything we can take into account as a feature to learn quicker, maybe viewers who liked the video had a better viewing experience than those who disliked it.

Quite simply the idea is that we start by recording the paths taken by every user who watches the video and with that aggregate data we can begin to find the paths most traveled. Then suggesting paths for future viewers, learning from their variations from that path previously learned and repeating the cycle.