Building ChatGPT-style tools with Earth observation

On March 25th, 2024, the blog post received 490 views and had 5 likes.

Just think about having the ability to inquire a chatbot if it can create a precise map that classifies crop farming in Kenya or whether the buildings in your vicinity are sinking. Moreover, envision receiving responses that are based on scientifically proven information obtained from Earth observation statistics.

The European Space Agency, together with its technology collaborators, is striving towards creating a tool that can transform information acquisition in Earth observation with the help of advanced artificial intelligence applications.

Digital Aid For Data Handling

Every day, the Earth observation provides us with a huge amount of important data, but it's challenging for people to extract the most valuable information from it. Luckily, artificial intelligence (AI) is there to assist us in dealing with massive and intricate datasets. AI can detect essential details and display them in an easy-to-understand format.

A project called I*STAR received funding from the ESA InCubed programme to create a platform utilizing artificial intelligence to track natural calamities such as earthquakes or volcanic eruptions. The purpose is to provide satellite operators with timely updates to plan data acquisitions for their clients with ease.

The SaferPlaces AI software, which received support from InCubed once more, generates maps of flooding zones for emergency crews with a combination of on-site measurements and satellite information. SaferPlaces played a critical role in examining flood damage in Emilia-Romagna, Italy during the previous year's flooding.

Over the past couple of years, Artificial Intelligence has made remarkable progress. The development of noteworthy tools like ChatGPT and Gemini has even astounded the specialists working in this domain. To make the most of this revolutionary invention and seize the possibilities provided by this technology, the logical progression would be to create a text-based inquiry system similar to ChatGPT and apply it to observe data of the Earth.

ESA is collaborating with different organizations in the space, computing, and meteorology sectors to create a digital assistant for Earth observation. This program aims to comprehend human inquiries and communicate with responses that resemble natural language.

It's not unexpected that there are several components needed to make a digital assistant. The first step is creating a strong foundation model to support it. This is like putting together a jigsaw puzzle.

Car Engine Roaring Beneath Hood

Artificial intelligence models operate by continuously developing through training, while conventional machine learning relies on providing the machine with comprehensive data sets that have been marked, typically by a person.

Let's talk about foundation models, they have a distinct way of working. Foundation models are types of machine learning models that don't rely on human guidance. They use massive amounts of diverse data that isn't labeled to undergo training. These models are versatile and can be customized to suit specific purposes.

The consequence is an adaptable and strong artificial intelligence system. Since they were established in 2018, basic models have played a large role in the revolution of machine learning, affecting various sectors and the general public.

The ESA Φ-lab is currently working on multiple projects aimed at developing fundamental models specifically for Earth observation-related activities. These models rely on data to offer insights on urgent environmental issues like preventing extreme weather events and identifying methane leaks.

The PhilEO project, which began earlier in 2023, has now reached its completion stage. A framework for its evaluation has been formed through the use of worldwide Copernicus Sentinel-2 data. The PhilEO model will also soon be released and made available to the Earth observation industry, the aim being to foster collaboration, foster progress, and assure the widespread verification of the resulting foundation model.

The picture displayed above depicts the Richat Structure, a distinctive characteristic that the PhilEO model has acquired the ability to identify autonomously, without any human assistance.

Various programs at the ESA are currently focused on creating a digital assistant that can assist users in processing natural language queries. This assistant would utilize models based on Earth observation data to provide the most relevant information in either text or image form.

The preliminary version of the Digital Twin of Earth initiative has just proven that its virtual helper model is capable of performing various duties that require different modes of operation, conducting thorough searches across several data repositories, including Sentinel-1 and 2, to compare and analyze data.

An upcoming ESA Φ-lab project, set to commence in April, will delve into the realm of natural language processing in order to extract and analyze verified information from text sources related to Earth observation. Additionally, the project aims to interpret queries from both experts and non-experts. The ultimate goal of the project is to develop a digital assistant that will operate seamlessly.

Giuseppe Borghi, who is the Head of ESA Φ-lab, shares that it is exciting to think about having an Earth observation digital assistant that can offer diverse insights from different sources. He adds that these initiatives indicate that there are essential components that must be established to achieve this goal.

Based on the highly positive advancements made with PhilEO and the earlier version of the virtual helper, I am confident that the upcoming undertakings will produce revolutionary outcomes in the upcoming days.