Internet of Things: Augmented Reality

Internet of Things:


Augmented Reality

The Internet of Things has growing momentum, both in terms of the underlying technology as well as the adoption by corporations, universities and across many industries. One of the primary applications of IoT is its use in augmented reality systems as can be seen in several examples included in this post. While AR is not the only application of IoT, it is an area where progress can be seen and measured directly. This is primarily because there are so many entities creating deliverable and consumable implementations. These bleeding edge AR-IoT models are starting to create momentum on mobile platforms like the iPhone and Android which have themselves contributed to the recent interest and growth of IoT.

The image to the left represents a simple implementation of AR that when extended shows the obvious implications for how these two technologies will come together. This picture was taken from my webcam which was able to superimpose a moving image on top of a thing, in this case the thing was a piece of paper with a glyph printed on it. I've included the link to try this yourself as well as a video that shows how it works. This is a simple but very effective method of enhanced communication.

Overview

In doing the research for the Internet of Things articles it became obvious that one of the most important driving factors is augmented reality, a term that is part of a broader definition called mediated reality. It isn’t really fair to just say augmented reality because there are other variations that will take advantage of the same technologies, these include machine vision, ambient intelligence, context sensitivity and of course virtual reality. When combined these technologies hold incredible promise to make almost every aspect our lives better, from education to entertainment to the quality of our health care and medical services.



We’ve all seen augmented reality in the movies, it has been the primary technology for several major motion pictures like Minority Report, Terminator 3 and even a recent John Mayer music video was littered with augmented reality. The Chevrolet Corvette has been using a HUD display for years now, but GM and others like BMW (and Mini-Cooper) are looking to extend the concept to help visualize the environment. A simple example is someone that has trouble driving at night, with an AR based HUD the windshield could actually highlight the curvature of the road, any objects in or near the road, etc. Another more common example is the first down marker used in televised football games that indicates where the offense needs to reach in order to get a first down.

Not all of the examples in this post relate directly to IoT but they are required to get a full understanding of how augmented reality technologies work, in most cases it will be obvious how these systems will be extended to include IoT initiatives. The intention of this post is to provide an overview of augmented reality systems and how they relate to the Internet of Things through the following topics:

    •    Definition and Related Technologies
    •    Common Terms
    •    How it Works
    •    Applications
    •    Additional Resources
    •    Relationship of Augmented Reality to the Internet of Things
    •    Summary

That should provide a good overview of not just augmented reality but how it is related to the Internet of Things. When combined these two technological revolutions will change our society and our lives dramatically and forever. We are in the early stages of development but progress is being made at an increasing pace and just like the Internet of Things, augmented reality is being driven as much by citizen scientists as it is by the large corporations and universities. Many of the fundamental development tools are available under an open source license which is helping to drive software development into a much larger audience of technologists, this momentum continues to increase as more and more tools become available.

As usual I've included links throughout the post that provide clarification, more details or just additional information about any particular point being made or topic being discussed. The post also includes extensive videos and images that have been embedded to present a visual description of the applications discussed, after all these technologies are primarily visual in the way they are and will be consumed (although not exclusively). Finally, a section dedicated to additional resources has been included to provide access to details or other content and information that may or may not have been covered in the post.+

Definition: Augmented Reality

The basic definition says that any direct or indirect view of a real world situation that has been overlaid by computer generated imagery. This imagery is usually generated in real-time and can be applied to real world objects or can include virtual objects. The implementations have taken the shape of goggles, computer screens, glass surfaces and more recently contact lenses. Most augmented reality systems are based on a marker, tag or glyph system that identifies, to the computer, an object or location that can be augmented. The objects can be marked using any one of several different methods including RFID chips or visual markers that can act like bar-codes.

Some technologies can do visual recognition of objects which eliminates the need for tagging objects, but this is a processing intensive operation. Face recognition is the most common type of complex object recognition used by simple AR systems, but this changing because of, believe it or not, airport security systems that have the capacity to recognize all types of objects including but not limited to weapons.

There are other forms of mediated reality that are similar but represent a different focus. Virtual reality is a commonly known technology that describes a landscape generated completely by machines. Just like reality the virtual world can be augmented as well, this concept is called Augmented Virtuality. AV is where the computer generated landscape is augmented just as a real world landscape would be, we've all experienced this when playing video games, especially the first-person shooters like Halo and dozens of others.

How it Works

The concept of augmented reality is actually not that complicated, especially after many of us have been conditioned through video games and movies to understand what applications we should expect and how they might be used (see movie: Minority Report). The video games, especially first person shooter games like Halo and BioShock, provide a pretty good representation of some of things we can expect like HUDs and automatic object identification. This post will focus on three variations of the augmented reality theme:

1. Marker based augmented reality: This is what is happening today, real-time processing of reality on PC based webcams and mobile devices that can recognize certain markers and overlay images or video over them. Most of the simpler but still powerful examples of this can be seen all over the Internet, many work by doing nothing more than printing a glyph (marker), turning on your webcam and putting the glyph in the webcam's field of view. The current focus is using glyphs as the primary method of marking an object that can be augmented, in the future this will be expanded to include RFID and all the other tagging methods used by the Internet of Things. In this regard both IoT and AR have the same fundamental requirement of object identification.
2. Markerless based augmented reality: Today, this type of identification is typically done using a combination of GPS and compass bearings on mobile devices like the iPhone, Android and others. This is a growing area of interest with several applications already available for the primary mobile platforms, but the technology doesn't provide identification on a granular scale so it has limited applicability. This is changing however, as software becomes more and more mature and the sensing technologies extend their range and reduce power requirements this will become the dominant form of input to AR systems.
3. Augmented vision: This represents the ultimate form of augmented reality, the ability to see both the real world and any annotations without managing a separate device like a cell phone or a computer. Sure, the goggles exist but are cumbersome and don't exactly make an appropriate fashion statement, but these are fine in many commercial and industrial applications. There is short term hope however, nanotechnology has already established augmented vision in the form of a contact lense and while it isn't ready for public consumption it does prove the technology can be built.

Both marker based and markerless systems are in place today and continue to evolve and mature, with the quality of sensors provided in most cellular devices there it no reason not to continue to pursue both.The basic marker based augmented reality can be defined by the following five step process, detailed below:

The first step is typically accomplished by placing a printed glyph (marker) in front of a webcam, this is of course after identifying the site and what software needs to be run, sometimes nothing more than a browser running flash will work, but in some cases a custom download is required. In either case, the optics are turned on and trained on the glyph. The next step is for the software to identify, by reviewing the video stream, where the glyph is located. The software will then determine the orientation and attenuation of the glyph(s). Once this is complete, the object to be overlaid (video or images) is positioned and oriented according to where the glyph is in the stream. The final step merges the two streams together creating a seamless (hopefully) rendering.

Since orientation and position of the glyph can be changed in real-time it is important that the objects being overlaid into the stream are rendered in three dimensions, otherwise only a single perspective would be useful. This is true regardless of the method of marking being used. 

The markerless model is similar to marker based tagging but slightly more complicated as it includes calculations from GPS and compass sensors in addition to the video feed. This sensor data is then combined and searched to determine what objects can be recognized and augmented. For example, knowing exactly where I am and what direction I am facing enables the video feed to be searched with more precision, looking for specific items that can be framed and referenced based on GPS coordinates. The video on the left shows a completely different approach cooked up by a student at MIT.

There are combinations of these models and they are not limited to the implementations described above, as previously mentioned video search and recognition software is getting increasingly mature, depending less and less on any type of marker and more and more on software processing capacity. The ability to identify most important objects with software will mark an important transition for AR. The airport security systems have driven much of this requirement in trying to determine what packages contain by just software looking at x-ray images. The video to the right provides an alternative model of augmented reality, one that nicely integrates real world activities into our digital lives, by the way this is the same guy from MIT that created the technology displayed in the video above.

One final point to make about markers is the importance of them and how both approaches are required. The problem with markerless tagging is that it only supports one-way communications between the objects and the viewer whereas marked things can, with the right hardware, interact with the viewer. This is clearly a more powerful model since there are computing resources involved and a bidirectional communication channel exists between the viewer and the object. Wouldn't it be nice if my cell phone could query a book and retrieve its content without the use of the cloud? This is possible with nanoscale components!

Common Terms

While the above definitions do not represent the extent of variation in this field, they do represent both extremes with augmented virtuality being somewhere in the middle. This is one of those areas of technology that has its own extensive and proprietary terminology.

The section below describes some of the other variations and combinations of terms used to represent these technologies. As with other areas of this post, links have been included if you want to dive deeper into any of these definition or related topics:

    •    Ambient Intelligence: Environments that are sensitive to the presence of people.
    •    Machine Vision: Application of computer vision to manufacturing and industry.
    •    Context Sensitivity: Automatically identify the current set of parameters required to make decisions.
    •    Ubiquitous Computing: Information processing capabilities exist in most devices and things.
    •    Virtual Reality: A computer generated landscape or version of reality.
    •    Augmented Reality: Real world environment agmented (overlaid) with computer generated imagery.
    •    Mediated Reality: Add, substract or otherwise manipulation the perception of reality.
    •    Augmented Virtuality: The marriage between augmented reality and virtual reality.
    •    Virtuality Continuum: Sometimes called the reality continuum, represents a scale binding virtual and actual reality.
    •    Wearable Computer: Exactly what it sounds like, part of the ubiquitous computing paradigm.

There are dozens of other terms that are or can be applied to mediated reality, whether it is the virtual or augmented type. If you want to find out more about any of these variations either check the links above or refer to the additional resources section that contains a curated list of links to blogs, websites, videos and other information. This is a very broad topic area that is being pursued by multiple industries using multiple approaches, all of which work to varying degrees and all have implications that extend well beyond the Internet of Things.

Applications

The applications for augmented reality are practically endless including everything from education through applying medical care and services. Some industries are further along than others but almost all have at least a toe dipped in the waters of augmented reality. In many cases these are localized, proprietary and very specific applications of the technology serving a particular purpose. This is a normal part of a market's evolution, it generally takes several iterations of a technology to become mature enough and general enough to have  applicability outside of the original intention.

The impact augmented reality will have on education will likely be substantial and will ripple across all areas of society including not just schools and universities but manufacturing facilities and operations centers. These augmented reality applications will ultimately change entire industries, rendering many irrelevant and useless while at the same time creating a whole new paradigm for new companies and industries to take advantage of. Again, a typical outcome of technological evolution is that it leads to a resorting of industry where some companies are eliminated and others created.

Education

This particular area spans every aspect of human existence and applies to all other industries and applications for this technology. Being able to provide intuitive and easily consumed educational concepts is critical for any society and any endeavor within that society. This is especially true in the age of the Internet but will likely become more important as time moves on and change happens at an even more rapid pace than it does now. In fact, many argue that without a substantial improvement to our education and training systems people will not be able to keep up with the pace of change. In the past, educational systems have been able to catch up with the revolutions created by technological advance, but this is less and less likely as the pace of change increases at an almost exponential rate.

Fortunately, this is the area of augmented reality that is receiving considerable attention and has seen the most improvement. Educational resources based on augmented reality have taken several forms including interactive books, real-time process simulations and visualizations as well as a number of other areas. Each industry is trying to determine how to best use this new capability whether it is the development of educational, training or instructional implementations most realize the importance of these materials. Medical and military AR applications have been used for training and instruction and are at the bleeding edge of this technology, providing both training of difficult procedures and instruction for on-field response to various trauma.

It is likely that this area remain one of the most active in terms of research and development, skills developed in these areas should lead to lots of high demand opportunities within the next several years. Educational resources developed using a combination of these technologies should provide a substantial improvement over any of the traditional methods previously used. This is important for two reasons; allowing humans to interact with an increasingly complex environment and allowing humans to deal with the increasing pace of change both personally and professionally.

Medical Industry

The medical profession stands to gain quite a bit from the implementation of augmented reality in their field, especially surgeons. One example is similar to flight training simulators where a surgeon could effectively practice a surgical procedure until they can do it perfectly. This simulation could be done with the actual patient information by combining sensor input from MRIs, CTs and even ultrasounds onto a platform that can recombine it into a 3D virtual person. 



Eventually, the real-time data acquired from sensing devices will be combined and superimposed on the real patient, guiding surgical procedures with precision and accuracy. This type of technology would increase the effectiveness of minimally invasive surgery by providing the surgeon with a complete view inside the patient, this is the biggest problem with these types of procedures today. The ability to have an unobstructed view inside the would inevitably lead to more surgical procedures that could be performed as minimally invasive.



There are several groups that are actively pursuing the ability to combine non-invasive sensor technology like MRIs and CTs directly onto the patient allowing physicians to actually have x-ray vision. The various imaging technologies continue to increase in resolution with each iteration bringing physicians closer to visualizing low-level biological function in real-time. At UNC Chapel Hill, a research group has demonstrated the ability to use ultrasound to create a 3D image of the fetus in real-time and display it directly on the patient’s abdomen, this allows caregivers to see the actual baby moving and kicking inside the womb.

Manufacturing

There are several ways that augmented reality can play a role in the manufacturing environment. Educational resources will be used to explain complex machinary to operators, mechanics and technicians. It will be used to visualize real-time processes and make adjustments to them on the fly. The use of goggles will become commonplace for certain types of operations like preventative maintenance, troubleshooting and repairs. Boeing is experimenting with an augmented reality process that guides technicians on how to connect massive wiring bundles.

Annotation and Visualization

These are fundamental concepts and like education apply to almost every implementation of augmented reality. The ability to identify an object and the provide an interface to the information about and attached to it, including any private or public annotations. The annotations might be reviews, comments or any number of discrete pieces of information that can be associated with people, places and things.  

Visualization is a way that augmented reality can help highlight what are otherwise difficult to see objects. A classic example is night driving where it is more difficult to see the curvature of the road, visualization can augment the view of the road and overlay bright lines that highlight the curvature and even other objects on or near the road. These overlays will be projected onto the windshield as part of a heads up display (HUD). These techniques are used today by the crew of the space shuttle to outline to curvature of the shuttle bay during operations, it is also used by jets when landing on aircraft carriers at night.

Ultimately, the hope is to create a sort of bionic lens that can be used to augment vision without goggle or other bulky devices. This is actually becoming a reality as a research group at the University of Washington was able to construct such a lens and prove that even in the early stages of development the lens can perform rudimentary functions. The lens uses nanoscale LEDs that act as the screen and solar energy to actually derive the power it requires to operate. The group is now trying to add more pixels (LEDs) to the device as well as a wireless communication capability so it can interact with a cell phone or other wireless, Internet connected device. See the video on the left to get a perspective on how the contact lens might work.

Entertainment

As you can imagine, this is a very popular area of engagement between the real world and the implementation of augmented reality. The recent smash hit movie Avatar has a web site that is co-sponsored by Coke where an augmented reality commercial is available to interact with, you can print the glyph instead of having an actual Avatar Coke can. There are several other companies leveraging the success of Avatar and augmented reality including Mattel and McDonalds. This is only one of several different forms of entertainment using AR, recently John Mayer released a video that is littered with augmented reality. The entertainment industry always finds a way to exploit the new technologies, typically by helping to drive the trends of adoption and use. The trend of augmented reality seems to be firmly seated in Hollywood.

Other

There are million other places where these capabilities can be leveraged but trying to describe them all is silly, at some point it becomes a broken record because the basics don't change from industry to industry, just how they are implemented. Basically, augmented reality is being applied in almost all industries in one form or another,  the most common use is education and training so far but this will likely change and refocus energies on entertainment, advertising and other forms of communications.

Additional Resources

Relationship of Augmented Reality to the Internet of Things

There is a direct and fundamental relationship between these two technologies that is primarily based on mutual success. It turns out that the components fundamental to one are also fundamental to the other, and not just technology but also adoption and use. Because they share so many fundamental components it is hard to separate one from the other in any sense beyond the name. The Internet of Things will most likely use augmented reality as the primary mechanism for consumption and interaction of the benefits it promises to provide.

Tagging or marking objects and locations is a fundamental requirement for both technologies, being able to identify objects in the environment is a key to their successful implementations. In addition, both are very dependent on quality components and sensors like a GPS, compass, video camera and microphone. Without tagging systems, sensors and small electrical components neither AR or IoT would have a chance of providing any practical value in the real world.

I think it is fair to say that the success (or failure) of augmented reality based applications and the Internet of things are co-dependent, largely because of the fundamental requirements that both share. Right now at least both are benefiting from the relationship as they jointly evolve and generate momentum for each other. Both are combining to create an Internet that has critical senses like hearing, seeing and speaking.

Summary

The world of augmented reality has come a long way in just the last couple of years, helped along by the same fundamental components that have been driving the Internet of Things. The continued use and growth of cellular and mobile devices  is driving both of these revolutions, this is likely to accelerate over the next decade as the components continue to come down in price. The nanotechnology revolution dovetails nicely because it is providing the fundamental components required, things like radio transmitters and various types of sensors are smaller and require less power then ever before.

Society has been conditioned and now expects these types of technologies, sooner than later. This is obviously one of the most critical driving factors, adoption. It is likely that adoption of newer technologies continues at a pace consistent with new developments, this has been the trend over the last decade. Apple and Google are providing two of the key platforms where these innovations will take place, positioning both companies as important players in the areas of augmented reality and the Internet of Things.

The next few years should see rapid improvement in the fundamental technologies and related business models in augmented reality, this is similar for the Internet of Things. Both technologies are moving hand and hand into the future, depending on the same or very similar hardware and software to drive their function, value and ultimately adoption. I think one or more then one of the major software companies will dive into this and turn it into a real market. Apple, Google, Microsoft or IBM are all sitting on fringe right now and could easily step in to substantiate and dominate the space.

Word Cloud of Post below: