5 Chapter 1 Information Technology Advances, Remote ID,[1] & Extreme Persistence ISR [Ryan]

Student learning objectives.  After reading this chapter, students should be able to do the following:

• — Identify, describe, and explain the advances in technologies that affect unmanned system design and use
• — Discuss the impacts of technology advances on the ability to remotely identify individuals
• — Discuss how energy availability affects the range and duration of unmanned system operations

The Emerging World of Unmanned Vehicle Uses

As noted in the preface to this book, unmanned technology has undergone a growth explosion.  Innovations in production, sales, testing, specialized designs, and uses continue to push the adoption of unmanned systems into new markets while expanding its use in existing markets.  The improvement in the underlying technologies, such as cameras, have enabled many new uses.  Further, use in one area can inspire use in another area as knowledge about capabilities spreads.  The first types of jobs that are being tasked to unmanned vehicles are typically the ones that are dangerous or relatively expensive to use humans to execute.  The next types of jobs being assigned to unmanned vehicles are the repetitive but simple tasks that are less expensive to execute without humans.  Examples of some of the emerging uses of unmanned systems include the following:

• Governmental uses, such as for military purposes and population control. The use of unmanned systems by military forces includes bomb disposal robots, surveillance drones, and battle damage assessment (Page, 2020).  Police forces use them for “mass surveillance, crime investigation, search and rescue operations, locating stolen goods, and surveying land and infrastructure.” (Electronic Frontier Foundation, 2017)
• Emergency services, including first responder support and humanitarian relief. These types of applications assist emergency personnel in finding victims, surveying disaster areas, and getting relief supplies to people who would otherwise be unreachable. (CB Insights, 2020)
• Construction and infrastructure. Drones are being used to not only monitor the progress of construction (Burger, 2019) but also to move supplies around construction zones.  An emerging use of drones is to actually construct buildings: a proof of concept experiment in Switzerland used drones to “lift and stack thousands of polymer bricks… to create a geometric structure nearly 10 meters high” (CB Insights, 2020).
• Crime. From transporting drugs and contraband into prisons and across borders to delivering bombs and spying on people, unmanned systems provide a cost effective and risk reducing capability to determined criminals. (Swales, 2019)

Predictably, some of the uses are hostile to one or more parties.  Of course, hostility is in the eyes of the beholder: a criminal views law enforcement use of unmanned surveillance technology as hostile to their interests; a pro-democracy protestor views population surveillance as hostile and adversarial.  The point is, of course, that the technology itself is agnostic as to usage and that it is growing in leaps and bounds, with innovations in both the underlying technology base and in the use of the technology coming seemingly every day.

In order to understand how the uses of unmanned systems can evolve and adapt, it is necessary to keep tabs on how the component technologies – sensors, communications, controls, etc. – are evolving.  This chapter provides an overview of these advances from the perspective of unmanned systems.  Many advances in technologies and uses have occurred, with some of the most interesting being in the marrying of information technologies – imagers, signal sensors, etc. – with advances in aeronautics, energy sources, and engine technologies.  In this chapter, we will look some of the more interesting technology changes and explore some implications.

Information Technologies

When talking with people about unmanned systems, it is sometimes surprising to realize that they do not fully appreciate the fact that few systems are fully autonomous.  When something is unmanned, it just means that the “manning” is outsourced.  But what is “manning” and what is “outsourcing”?

“Manning” implies a lot of things, most specifically that a human is in the operational loop in the same location as the equipment being operated.  When a piece of equipment is “unmanned”, then one or more of those elements is different.  It may be that a human is still in the operational loop, but at a different location.  It may be that some aspect of human intelligence has been automated and integrated into the equipment.  Or it may be a combination of those.  Outsourcing is when one or more tasks that one person would do is given to another entity to perform.  Aspects of human labor are most easily outsourced to machines but could also be outsourced, when possible, to distant people.  Aspects of human intelligence can be outsourced to smart machines, to distributed groups of people, or to companies.  Therefore, when we say that “unmanned means that manning has been outsourced”, we are really saying that some aspects of human labor and cognition have been separated out and assigned to some mix of distant humans, distant machines, and on-premise technologies.

The operations of any unmanned system is complex.  As noted in Nichols et al (2020):

A UAS can’t fly (very far) if it doesn’t have internal systems to parse received instructions, make decisions based on sensed data, and control its onboard systems.  The internal systems can be thought of as the internal nervous system of a UAS.  Sensed data is collected and may possibly undergo some preprocessing, prior to being transferred to a decision support system, a suite of AI support elements, or external communications for relay to other UASs and/or command and control elements, such as an airborne control system or a ground control system.  The internal systems interpret and instruct navigational control, mission execution, and propulsion control.  When emergency situations occur, the internal systems execute preprogrammed options, which could include autonomously navigating to safe zones or self-destructing.  The internal systems also monitor the health and welfare of the UAS according to the instrumentation included onboard.  This may include fuel level monitoring, damage assessment, and interference detection.  According to design, the internal systems may relay information continuously, on schedule, or in emergencies.

The separation of human labor, cognition, and equipment into disparate pieces means that a concomitant need for collaborative technologies becomes important.  The most obvious collaborative technologies are communications – exchanging data, commands, and responses.  Other types of collaborative technologies that need to be considered are those that prevent adverse interactions between system components, environmental sensing and reaction technologies, and control guidance technologies.  All of these enable the remote operation of an unmanned system.

When considering the operation of anything that is unmanned, be it an earth-bound robot or a high-flying weather balloon, it is very useful to think about two things: where the intelligence associated with the operations is located, and how the intelligence is divided between elements.  The process of automation is fundamentally the process of taking elements of human intelligence and replicating those elements in machines.  Advances in automation follow the development of technologies that allow increasing distance between humans and their tools, but also follow a fairly predictable cycle of development, use, integration, and innovation.  Successful innovation can both restart the cycle and spawn separate cycles.

In order to explore advances in information technology as applied to unmanned systems, we will look at the several general categories of technologies and how they are poised to affect the future of unmanned systems.  When appropriate, examples of real-world implementations will be provided.  (The use of these examples does not constitute endorsement: they are simply examples of what is going on in the world.)  The general categories of technologies covered are:

• Internet of Things (IoT)
• Artificial Intelligence
• Advanced Manufacturing

In addition, a short look at energy advances is included, because without portable and sufficient energy, unmanned system operations are limited.

Internet of Things (IoT)

The term “Internet of Things” was coined to describe the network of everyday items (such as thermostats, toasters, and pet collars) that have been enhanced with information technology and enabled with communicative capabilities.  The simplicity of the term belies the extraordinary complexity of the reality.  Billions of devices sharing data over the internet means that tons of very granular data can be combined to create new knowledge.  Moreover, the direct connection to each device enables a level of control that transcends the individual component.  Within the last decade, major advancements in two fundamental technologies, sensors, and wireless communications, have led to an explosion of the IoT.  According to SAS, there are “127 new IoT devices connected to the internet every second” and there will be “more than 150 billion devices connected across the globe by 2025”.  Further, “the global datasphere will grow from 33 zettabytes in 2018 to 175 zettabytes by 2025.” (SAS, 2019)

The phenomenon of linking many small devices over very large networks is one aspect of the IoT.  There are also geographically smaller implementations of interconnected devices that create what can be effectively considered a swarm of minimally intelligent elements collaborating for a larger purpose.  One such implementation has been called an “internet of bodies” (IoB), where human bodies wearables and embeddables are connected to a network.  The initial purpose of IoB has been to augment health treatments, diagnoses, and monitoring, but once human health and behavioral data is being shared and accessed via the greater Internet, many possibilities emerge, including the merging of human and machine capabilities over great distances.

Near Term IoT Technologies

According to Strativerse, a collaborative research effort that “analyzed the capabilities of current technologies and their likely trajectory” (L’atelier BNP, 2020), there are several technologies that are at technology readiness levels  [2]  (TRL) 7 or higher that are worthy of consideration.  These include data marketplace, simultaneous localization, and mapping (SLAM), and digital twinning.  Those near-term technologies are already being used in some areas but could be more widely adopted in the coming years as the infrastructure support for them expands.

Data marketplace is the term given to a service that allows people to buy and sell data.  Buying and selling data is nothing new but the adoption of marketplaces to connect buyers and sellers has made the practice more convenient and efficient.  As noted by Jeremiah Smith, data marketplaces provide the following advantages:

• Crowdsourcing:by making self-serve data selling a reality, they provide the solution to move away from inaccurate/expensive single-source data.
• Aligned incentives: data owners/collectors directly benefit from keeping data in structured form and making it available to others.
• Standardization: by design, a marketplace defines a common data model and interface for buyers and sellers to exchange data.
• Fairness: instead of a having a central authority pricing data, providers can set their own prices while consumers can choose who they buy from. (Smith, 2018)

How might this affect unmanned systems?  A big effect might be increased use of unmanned systems by individuals: smaller operations are incentivized because it makes collecting and selling data, even as a single person operation, possible.  This incentive makes it attractive to collect more data and to compete for buyers on both price and accuracy.  Over time, this could translate to aggregation of data providers, making the business case for operating large fleets of unmanned systems for the purpose of purely commercial data collection.

Simultaneous localization and mapping (SLAM) is already in use by some unmanned systems and should be expected to expand to many more.  Imagine being taken into an unfamiliar building and being asked to draw a detailed and accurate map.  You don’t know where you are, you don’t know the dimensions of any of the rooms or even how many there are, and you don’t know if there are stairs or elevators.  When you start drawing the map, by measuring first what is around you and then moving outward, you are doing simultaneous localization and mapping.  Asking a human to do this is tricky but ultimately doable, because of the experiences that the human has accumulated over a lifespan with navigating and mapping.  Teaching a robot to do this more difficult because the problem is inherently paradoxical: in order to know where it is, the robot needs a map, but in order to make a map, the robot needs to know where it is.  Furthermore, as it moves, it does not know where it is going.  (Martin, 2019) (Burgard, Stachniss, Arras, & Bennewitz, 2020)

The benefits of getting SLAM right are pretty obvious: drop an unmanned system into a danger zone and let it figure out how to navigate.  Send a robot to another planet and let it find its way around.  Launch an unmanned underwater vehicle to explore a deep canyon.  Send a drone into a cave system to explore.  As the technology matures, it will increasingly be used to assist unmanned systems in navigation, as well as provide important real time improvements in the quality of data about our physical environment.

Digital twins are highly detailed computer models of actual systems, which are different from other models in that they are continually updated with data from the actual system so the model reflects the current operational status accurately.  The key to successful implementations of digital twins is the data feed between the system and the twin: the digital thread. The distinction between the twin and the thread is time:  “the digital twin is the current representation of a product or system, mimicking a company’s machines, controls, workflows, and systems. The digital thread meanwhile is a record of a product or systems lifetime, from its creation to its removal.” (Miskinis, 2018)

The benefits to unmanned systems lie in the ability to exploit the real time aspects of the twin in order to detect and react to challenges faster.  The thread provides a way of analyzing the longitudinal aspects of the system.  For example, if “a vehicle has an accident due to the fault of the system such as unplanned acceleration, the digital thread through having traceability across the lifecycle of a vehicle, will be able to identify the issue.” (Miskinis, 2018)

Longer Term IoT Technologies

Atomic scale storage is an experimental approach to encoding information in atoms.  It is currently at TLR 4 and can only be done in a laboratory.  The promise of this technology is intriguing: “the storage density of this memory is 500 times larger than that of state-of-the-art hard disk drives.  If perfected, this technology could enable the manufacturing of memory cards with capacity of 62 terabytes.” (L’atelier BNP, 2020)

A synthetic doppelganger is a “realistic robot powered by advanced artificial intelligence with the ability to emulate one’s personality and subsequently substitute for them.” (L’atelier BNP, 2020)  The idea here is that a “human brain would be scanned and digitized, with the resulting content being fed into an artificial neural network.” (L’atelier BNP, 2020)  The application to unmanned systems would be in using the synthetic copy to operate in places where human bodies are not well suited, such as space or underwater, with the full learning, deciding, and personality cues of the original human.  This technology is at TLR 3 and faces many challenges before being realized. (L’atelier BNP, 2020)

Artificial Intelligence (AI)

The growth and promulgation of both weak and strong AI is already evident in many aspects of our lives.  The integration of AI into unmanned systems holds a lot of promise.  As noted in Nichols (2020):

… the increasing miniaturization of electronic components, the incorporation of alternatives to electronics, such as optics, and the development of special purpose processors have and continue to revolutionize the ability to squeeze capabilities into a small size form factor.  Size reduction has a lot of advantages: it can mean lower power requirements, faster execution of computational cycles, and less heat generation.  It can also have some inherent disadvantages, including less robust physical components.  Protecting advanced microelectronics from directed energy attacks, for example, can require significantly increased shielding, which can in turn affect overall energy requirements for flight operations.  In mission situations where energy efficiency and UAS maneuverability are important, trade offs need to be considered in overall system design.  However, great strides have been made in both the development of specialized processors that execute AI-like capabilities and the integration of those processors on common chip sets.  Integration of multiple special chips in a system can provide a marked improvement in on-board intelligence (Morgan, 2019).

The integration of advanced automation, including AI, into UAS architectures can be thought of as having several faces.  First, decision support systems with pre-programmed rules of engagement can be embedded onboard the individual systems.  Next, specialized AI processors can be included as well.  Naturally, more complex AI and decision support solutions can be implemented that rely on backend (either terrestrial or airborne) processing for the heavy computational lifting.  Finally, all of these can be integrated together. (Nichols, et al., 2020)

There is an enormous amount of progress being made in realizing applications of AI.  Three of the emerging technologies are already being incorporated into unmanned systems: facial recognition, gait recognition, and gesture tracking.

Facial recognition has gained a bit of notoriety because of some governmental applications.  The use of facial recognition capabilities in unmanned systems is useful for threat recognition as well as for commercial purposes, such as urban planning.  Understanding who goes where in a city and when can be extremely important information for city management purposes.

Abuses of human rights, however, have made it a controversial capability.  By 2019, 64 countries were using facial recognition for surveillance purposes. (Feldstein, 2019)  China in particular has made extensive use of facial recognition in many sectors, including “from catching criminals in huge crowds to detecting and shaming jaywalkers to deciding whether someone can get an extra square of toilet paper in a public bathroom.” (Samuel, 2018) This technology has been used in unmanned systems, with a new level of secrecy and disguise has been reached through the development of robotic birds for population surveillance. (Chen, 2018)  “The drones have wings that flap so realistically they’re difficult to distinguish from actual birds. In fact, animals on the ground often can’t make the distinction, and even real birds in the sky sometimes fly alongside the drones.” (Samuel, 2018)

Gait recognition is less widely proliferated but of great interest to augment facial recognition and other identification methods.  Gait recognition is based on analyzing how people walk.  Even those who have very similar faces move differently.  “One promising extension of gait recognition is to use inputs from multiple or moving cameras … (e.g., cameras installed on drones) [to] actively detect people behaving suspiciously by capturing pedestrians from different viewpoints.” (Shigeki, Okura, Mitsugami, Hayashi, & Yagi, 2018)

Gesture tracking, which is a technology that captures how people move, enables a new type of interface for computer system control. While the early implementations of gesture-based controls relied on instrumented gloves and other wearables, the use of computer vision and other sensing systems has freed the human from having to don equipment.  “Gesture-based technology is already in place and commonly used (e.g., public buildings, public restrooms) without special instruction required for effective use. A common example of a well-designed gestural command is the use of hands to “wave” to activate (e.g., public bathroom faucet).” (Elliott, Hill, & Barnes, 2016) The use of gestures to control unmanned systems can enable a more intuitive and rich interface with the systems. “Navigating and controlling a mobile robot in an indoor or outdoor environment by using … hand gestures offer some unique capabilities for human–robot interaction inherent to nonverbal communication with features and application scenarios not possible with the currently predominant vision-based systems.” (Stancic, Music, & Grujic, 2017)

Advanced Manufacturing

From creating new materials to creating new ways to use materials, manufacturing is being affected in ways that change many aspects of system use.  Two technologies in particular are having impacts on the use of unmanned systems: 3-D printing and auxetic material.

3-D printing can be done in many different ways, such as adding material incrementally to create something or incrementally removing material to create something.  Swarm 3-D printing occurs when many small printers are connected together in a swarm.  Each part of the swarm is assigned a small task.  The collaboration makes 3-D printing of very large projects possible. (L’atelier BNP, 2020)  The combination of the swarm approach to 3-D printing with unmanned systems opens up a wide variety of potential applications, including creating structures in remote or inhospitable areas.

Auxetic materials are those that can “react to changes in their physical environment” and which can act as sensors by virtue of harnessing that change. (L’atelier BNP, 2020) Auxetic materials act differently than normal materials: they get “wider when stretched and narrower when squashed.” (Mir, Ali, Sami, & Ansari, 2014)  They also “possess attractive acoustic properties, and it is found that at frequencies up to 1600 Hz auxetic forms of polymeric and metallic foams possess enhanced acoustic absorption.” (Mir, Ali, Sami, & Ansari, 2014)  Current applications include smart bandages, which act both as sensor for detecting when medicine is needed and as protection for wounds.  The integration of auxetic materials into unmanned systems for enhanced resiliency and acoustic absorption is intriguing.  There is also an application of these types of materials to create a new type of prosthetic, such as a gripper, that is “smaller, more energy efficient and more puncture resistant.” (Chin, 2019)

Energy Sources

One never-ending challenge for automated systems of all types is energy access and availability.  Approaches to managing this challenge include reducing energy usage, improving energy efficiency, and creating power sources that have a higher energy density.  Lighter weight materials in unmanned systems contribute to energy management, as do the incorporation of renewable energy sources in systems (such as solar panels).  Advances in energy storage technologies, such as batteries, are being made as well.

Nuclear energy is contributing to small and portable energy applications.  A research team looking to “develop an energy source that is both small and light, and produces more power for a longer time” (MaterialsToday, 2009) created a battery that uses radioactive material.  The use of advanced materials for sealing and shielding the battery enables it to be used in a variety of applications. (MaterialsToday, 2009)  This work has continued in many different labs, with an expectation that commercial use of nuclear batteries could be achieved by the mid-2020s. “The space industry would also greatly benefit from compact nuclear batteries. In particular, there is a demand for autonomous wireless external sensors and memory chips with integrated power supply systems for spacecraft. Diamond is one of the most radiation-proof semiconductors. Since it also has a large bandgap, it can operate in a wide range of temperatures, making it the ideal material for nuclear batteries powering spacecraft.” (Moscow Institute of Physics and Technology, 2018)

Concluding Thoughts

The advances made in science and engineering are important for the future of unmanned systems.  The advances in information technologies enable more complicated operations but require more energy.  The advances in energy technology, both in energy density and in portability, open the door for more integration of advanced information technologies in unmanned systems.  The advances in manufacturing technologies and approaches create a rich future for unmanned systems.

While reading the rest of this textbook, keep these advances in mind and imagine how what you are reading will be changed, enhanced, or revolutionized in the coming decades.

References

Burgard, W., Stachniss, C., Arras, K., & Bennewitz, M. (2020). Introduction to Mobile Robotics Teaching Material. Retrieved September 12, 2020, from University of Freiburg: http://ais.informatik.uni-freiburg.de/teaching/ss12/robotics/slides/12-slam.pdf

Burger, R. (2019, August 15). 6 Ways Drones Are affecting the Construction Industry. Retrieved September 11, 2020, from The Balance Small Business: https://www.thebalancesmb.com/drones-affecting-construction-industry-845293

CB Insights. (2020, January 9). 38 Ways Drones Will Impact Society: From Fighting War To Forecasting Weather, UAVs Change Everything. Retrieved September 11, 2020, from CB Insights Research: https://www.cbinsights.com/research/drone-impact-society-uav/

Chen, S. (2018, June 24). China takes surveillance to new heights with flock of robotic Doves, but do they come in peace? Retrieved September 14, 2020, from South China Morning Post: https://www.scmp.com/news/china/society/article/2152027/china-takes-surveillance-new-heights-flock-robotic-doves-do-they

Chin, L. T. (2019). A High-Deformation Electric Soft Robotic Gripper via Handed Shearing Auxetics. Boston: Massachusetts Institute of Technology.

Electronic Frontier Foundation. (2017, August 28). Drones/Unmanned Aerial Vehicles. Retrieved September 11, 2020, from EFF Street-Level Surveillance: https://www.eff.org/pages/dronesunmanned-aerial-vehicles

Elliott, L. R., Hill, S. G., & Barnes, M. (2016, July). Gesture-Based Controls for Robots: Overview and Implications for Use by Soldiers. Retrieved September 14, 2020, from Defense Technical Information Center: https://apps.dtic.mil/sti/pdfs/AD1011904.pdf

Feldstein, S. (2019, September 17). The Global Expansion of AI Surveillance. Retrieved September 14, 2020, from Carnegie Endowment for International Peace: https://carnegieendowment.org/2019/09/17/global-expansion-of-ai-surveillance-pub-79847

L’atelier BNP. (2020). Strativerse. Retrieved August 20, 2020, from Envisioning Radar: https://atelier.net/strativerse/?pg=about

Martin, S. (2019, July 25). What Is Simultaneous Localization and Mapping? Retrieved September 12, 2020, from NVIDIA Blog: https://blogs.nvidia.com/blog/2019/07/25/what-is-simultaneous-localization-and-mapping-nvidia-jetson-isaac-sdk/

MaterialsToday. (2009, December 22). Small and powerful nuclear battery developed. Retrieved September 14, 2020, from Materials Today Energy News: https://www.materialstoday.com/energy/news/small-and-powerful-nuclear-battery-developed/

Mir, M., Ali, M. N., Sami, J., & Ansari, U. (2014, November 13). Review of Mechanics and Applications of Auxetic Structures. Retrieved September 14, 2020, from Advances in Materials Science and Engineering: https://www.hindawi.com/journals/amse/2014/753496/

Miskinis, C. (2018, November). What does a digital thread mean and how it differs from digital twin. Retrieved September 12, 2020, from Challenge Advisory: https://www.challenge.org/insights/digital-twin-and-digital-thread/

Morgan, T. P. (2019, November 13). INTEL THROWS DOWN AI GAUNTLET WITH NEURAL NETWORK CHIPS. Retrieved January 29, 2020, from The Next Platform: https://www.nextplatform.com/2019/11/13/intel-throws-down-ai-gauntlet-with-neural-network-chips/

Moscow Institute of Physics and Technology. (2018, June 1). Prototype nuclear battery packs 10 times more power. Retrieved September 14, 2020, from Phys.org: https://phys.org/news/2018-06-prototype-nuclear-battery-power.html

NASA. (2017, August 7). Technology Readiness Levels. Retrieved August 20, 2020, from NASA Engineering Technologies: https://www.nasa.gov/directorates/heo/scan/engineering/technology/txt_accordion1.html

Nichols, R. K., Mumm, H. C., Lonstein, W. D., Ryan, J. J., Carter, C., & Hood, J.-P. (2020). Counter Unmanned Aircraft Systems Technologies and Operations. Manhattan, Kansas: New Prairie Press.

Page, S. (2020, February 5). 5 U.S. MILITARY DRONE USES THAT MAY SURPRISE YOU. Retrieved September 11, 2020, from Sandboxx: https://www.sandboxx.us/blog/5-u-s-military-drone-uses-that-may-surprise-you/

Samuel, S. (2018, August 16). China Is Going to Outrageous Lengths to Surveil Its Own Citizens. Retrieved September 14, 2020, from The Atlantic: https://www.theatlantic.com/international/archive/2018/08/china-surveillance-technology-muslims/567443/

SAS. (2019, January 1). The Internet of Things. Retrieved August 20, 2020, from What is IOT?: https://www.sas.com/content/dam/SAS/documents/infographics/2019/internet-of-things-109082.pdf

Shigeki, Y., Okura, F., Mitsugami, I., Hayashi, K., & Yagi, Y. (2018). Directional Characteristics Evaluation of Silhouette-Based Gait Recognition. Retrieved September 14, 2020, from IPSJ Transactions on Computer Vision and Applications: https://link.springer.com/article/10.1186/s41074-018-0046-7

Smith, J. (2018, July 24). Data Marketplaces: The Holy Grail of our Information Age. Retrieved August 20, 2020, from Hackernoon: https://hackernoon.com/data-marketplaces-the-holy-grail-of-our-information-age-1211a6fec390

Stancic, I., Music, J., & Grujic, T. (2017, November). Gesture Recognition System for Real-time Mobile Robot Control Based on Inertial Sensors and Motion Strings. Retrieved September 14, 2020, from Engineering Applications of Artificial Intelligence: https://www.sciencedirect.com/science/article/abs/pii/S0952197617301975

Swales, V. (2019, November 3). Drones Used in Crime Fly Under the Law’s Radar. Retrieved September 11, 2020, from The New York Times: https://www.nytimes.com/2019/11/03/us/drones-crime.html