Tango - Teaching Geometry with AR

Project Status: App Idea

Requirements

This app would require a Google Tango device to run. Even though it is possible to develop an app with this idea now, it will become more fun and interactive as time goes on and Google Tango becomes more aware of not only the shape of the space around it but also begins to understand the objects found within that space.

If you are looking to buy a Tango device check out this post which links to the newest devices on the market.

The Idea

Make an app using Google Tango which helps people learn geometry by using the physical space of their environment and augmenting that space with visual aids to help understand the principles of geometry.

Classroom Use

Imagine being in geometry class and not only learning about points, lines, planes and angles on paper but also by seeing and playing with those ideas in physical space via augmented reality.

The Google Tango app could be used by teachers in classrooms to augment learning and understanding of concepts covered during the lecture.

Other Use

This app would also do well with adults and children that want to learn outside of the classroom. As in many of the educational apps already on the market, this app could provide content that covers many of the principles someone would cover in a classroom setting. The user could explore and learn in a self-paced manner.

Checkout the video

While this video from 2010 is not using Google Tango, it will give you an idea of how augmented reality can be used to teach geometry. Also it's not in English, but you'll soon see that doesn't matter.

Tango - Escape Room - Update

Project Status: Getting closer, but not yet Tango enough
Original Post: Tango - Escape Room

The Tango Idea

If you you haven't checked it out yet, and you are interested in Google Tango even in the slightest, I suggest you checkout my original idea for making a Tango escape room experience here.

If you are looking to buy a Tango device check out this post which links to the newest devices on the market.

Google Escape Room

Checkout the awesome escape room that people actually made, yes in real life, to showcase the usefulness of many Google products and the possibilities when they all come together. This is the short version in English:

The Extended Video

When they held this escape room event they actually live streamed it to YouTube. The video is forty-five minutes long and in French, you speak French and have forty-five minutes to kill right, lol. Don't worry, you can turn on the auto-generated English closed-captioning if you want to but it's not a requirement to still find this fun and interesting.

Here's the full video, minus the first 50 seconds of nothingness that I trimmed off for you, "Je vous en prie":

Next Step

Since the focus of this video was to showcase Google products, let's take it a step further and incorporate as much of this as possible using and showcasing Google Tango as laid out in the original idea here.

Tango - Escape Room

Project Status: App Idea
Update: Getting closer, but not yet Tango enough

Requirements

This app would require a Google Tango device to run. Even though it is possible to develop an app with this idea now, it will become more fun and interactive as time goes on and Google Tango becomes more aware of not only the shape of the space around it but also begins to understand the objects found within that space.

If you are looking to buy a Tango device check out this post which links to the newest devices on the market.

The Backstory

In the last few years you many have noticed lots of escape room adventure locations popping up. If you haven't been to one or haven't heard of them I suggest you search for some in your area to see what I am talking about.

Basically the idea of an escape room is that you start out locked in a room and must find a way to get out. The main door is locked and a series of puzzles must be solved for you to progress through a story-line and find a key or combination that will let you open the door to exit. Often within a given time limit.

You are often given some vague plot and it then becomes your job to find clues within your environment to solve several mini puzzles to as you progress through the plot and eventually, hopefully, make your escape.

The Idea

The goal would be to transfer a similar experience to your own room.

The Obstacles

The most difficult part of making a game like this work would be making it dynamic enough to work in nearly any given environment.

Masking Tango's Limitations

With devices overheating very quickly and compounding drift errors over time, longer games are not the best use cases for Google Tango at this stage in its life cycle. That's not to say that it isn't going to get better over time, but right now we need to develop apps that will work well for users today.

One clever way to overcome these kinds of problems is to mix up the gameplay so that extended periods of scanning and tracking the environment would not be common. As the users stumbles across some useful clues they can break away from augmented reality play and into what would be similar to mini games. Taking the known clues and solving puzzles not in space but in gameplay on the device. When they solve the mini puzzle, or when they choose to leave it for later on, they can pop back into augmented reality and begin exploring the environment again.

Summary

The difficulty level for developing this app is quite high right now, but it's nothing that a small team couldn't handle within a few months and then expand on over time. Eventually, as computer vision becomes more integrated into Google Tango and the average developer is able to identify objects in space on the fly, this will only get better and better.

Update

I recently came across a super cool Google escape room event. It's not quite Tango but it is all about what this project is. Check out the update and the video here.

Tango - I Spy

Project Status: App Idea

Requirements

This app would require a Google Tango device to run. The development skill level is quite a bit more difficult than previous app ideas posted here on this blog. If you are looking to buy a Tango device check out this post which links to the newest devices on the market.

The Idea

The gameplay is based off of the child's game often called I Spy. The app would see/scan/learn the room the player is in and use computer vision to secretly select objects within the environment. The game is to have the player try and guess which object the app is thinking of, much like the child's game often played in real life. Not only would this app potentially be fun to play, it could be used to help train computer vision neural networks.

Gameplay

When you open the app the game may take a second to acclimate, this is basically how Tango works now as it often says "Hold Tight," only with the added process that the app will have to try and use computer vision to quickly see an object in the field of view that it recognizes.

The very first object selected by the app each time it opens will likely be super obvious and easy to guess, which will serve several purposes. For one thing, the quicker it can find an object the faster the game will start. This saves from having to have the player scan large portions of the room before playing, We know that the player will naturally move the device around and we can scan the room during gameplay. However; if no objects are visible or recognizable right away, we may need to ask the player to slowly look around a bit with the device until this is solved.

Another benefit of this quick and easy recognition is it will help us to understand our player. If he/she is not able to quickly guess an obvious object within a small field of vision, then we will know to keep the object selection and gameplay easy for him/her as it is likely this is a child or someone that would have a hard time guessing in general. On the other hand, if he/she quickly identifies the object we will know that we can safely step up the difficulty when possible.

The app will need to provide a hint to start the player off such as, "I spy something red." The player can walk/look around and touch objects to guess if they have found the correct object that the app is thinking of. When wrong, more hints may be provided over time to help guide the player. If they player is having a hard time the app could even use visual feedback to show if the player is getting "warmer" or "colder" as the device is moved around.

Hints could be basic like "look down," "try higher," or more complex like "it is shiny," "it may be wooden," or "the object is soft and fuzzy." Furthermore; if it happens to be a pretty common object that the app understands clues could be something like this, for a door for example: "you use it," it swings," or "it's closed."

It is possible to do a little machine learning here. At the end of each round we could let the player point out if any hints were misleading or flat out wrong. Of course we could playfully apologize while noting that we are a simple minded AI character that he/she is helping to teach, promising to try and do better in the future.

When the correct object is selected the app could continue with the next object or by allowing the player to think of an object. Reversing the roles is quite a bit higher of a challenge for the developers but could always be added later on as a feature to the game.

Training Computer Vision

By reversing the roles, allowing the player to pick objects, we now have a nice opportunity to train our neural networks and make our computer vision results better over time.

There are several creative ways to allow the player to provide hints to the app, as it now must guess the object. One way is that we could start out by allowing the user to pick from a dynamic color palette specific to this environment. Essentially saying "I spy something the color..." by tapping this selection.

After that point the app could lead the player to objects is wants to guess and provide the player with three options to always select from being something like: yes, no or I don't know. "I don't know" could be worded many other ways but it's basically the option for what you are asking or guessing doesn't make sense, so please try something else or clarify. The app could continue to select objects or pose questions for the player to answer such as "Is it food?" The questions would always be geared towards a yes/no answer and would often teach us something about the object in question.

Other ways of interacting that allow the player to select objects would be to use speech recognition or providing a keyboard and an input box, but typing doesn't sound very fun in this game format and we are already making our workload hard enough without having to deal with speech recognition for what would often fall into the same three categories of yes, no or I don't know.

The Development

While this would be a really fun project to work on or be involved with, it is way over my skill level at the moment. Hopefully though someone will read this and take off running with the idea. Good luck to you and please keep me posted with the progress. Right now I am simply following tutorials and trying to get anything working with Tango. Let me know if you would like more clarification on this idea, I often tend to rattle off parts of an idea that make sense in my head but I lose people along the way by leaving out major portions of the inter-workings. 

Typeface Detection App for Font Finding

Project Status: App Idea

Problem

Sometimes we see the use of an interesting or unique typeface in print or in graphics and we would like to know which font that is exactly. This way we can download it or purchase it for our own use.

Workaround

The current way of tracking down a font is by searching through many fonts online hoping to randomly come across a matching or similar one. Another way is to try to track down the person who designed the graphics or print and attempt to ask this person or team directly.

Solution

I would like to use computer vision, such as OpenCV, and design an app that will not only recognize that there is text in an image/video but will discover the name of the font type used and link to it.

The app will likely be used over and over for some very common fonts and a small database can be made to handle this. By nature though the app will be used in cases of very obscure typefaces and possibly even ones that aren't fonts but the one-off designs used for specific applications.

When a font can not be determined it can be logged. The app user can choose to be notified if the font is labeled at sometime in the future.

Actually, if you have ever used the Shazam mobile app it would work in much the same way as this does but instead of music based data it would focus on fonts.

I would also like it to be a useful tool to those people that design fonts. This app could help them to sell and distribute fonts they own or to popularize ones they give away for free. If done correctly it could also help provide them with user interest feedback, in ways that may help them in designing more fonts. For example; if they see the unlabeled fonts users are searching for they may be able to develop similar fonts that those users may also like.

The tagging process, calculating the probabilities of type matches and even finding similar font types could each largely be done or helped by machine learning. The algorithms and bots could carry the largest parts and the most tedious portions of the workload while their human counterparts could guide them and help to improve the results they generate.

There are several easy to find and scrape font repositories on the web but it would eventually be interesting and useful to write bots to hunt down and include the more obscure ones out there.

FaceRig's Unveiling

Project Status: Complete

I have tons of ideas and I love to see when people are actually making them happen, like the guys over at FaceRig Software for example. It was just about two weeks ago when I was thinking about how cool it would be to have a webcam app that analyzes facial expressions and doesn't broadcast the source video but instead broadcasts a selected 3D character.


I like the fact that you can also change your voice to a voice that matches the animation, this is what I would have done also. These guys are doing pretty much everything with the project that I would do if I had the resources to do so.

Check out this video to see what they have so far:

I'm very glad to see that someone is already working on this idea and obviously has been for a while now. It'll be interesting to watch this project progress as they continue to work on it.