EMSA Runs a Tight Ship with Azure-Based Maritime Data Platform

Codit worked with the team at EMSA to configure its existing infrastructure with a hybrid integration platform using Microsoft Azure PaaS and in a few short weeks it was possible to configure the Azure solution, which could process data three times faster than the existing on-premises infrastructure.

The first port of call was setting up the infrastructure and connecting the Automatic Identification System (AIS) data flow. The team improved the data quality checks, for visibility into potential problems.

“Each EU member state or data provider supplies its own ID for every ship. To prevent duplicate IDs in the EMSA database, the on-premises infrastructure assigns each ship a unique, EMSA-specific ID, which unfortunately caused many anomalies in the tracking information,” explained Gillian Verstraete, Codit Azure Architect “We set up the new solution to examine the ingested data, proactively identify anomalies and missing data, and alert EMSA of any issues. The solution also now has an integrated dashboard so that the team can keep an eye on all traffic coming in, going out, and any messages that could create a problem.”

The Codit team also strengthened the solution’s back-end storage with a large-capacity Microsoft Azure SQL Database, which acts as a fast, first-line data provider for the most up to date (real time maritime picture) data to end-users.

The next challenge was to connect the database to the front-end application. The team leveraged Microsoft Azure API Management and built APIs that call and retrieve the position data requested by end-users and securely visualize this data in maps and other visual formats. Security was a prime consideration when developing this solution.

“Whenever a user requests information, such as a ship’s position, the solution must know if that user is authorized to see that particular ship,” says Joachim De Roissart, Codit Business Consultant and Project Manager, “We ensured this security in a way that doesn’t slow down the process for users, seamlessly integrating the solution with the EMSA IDMV2, a service that exposes the EMSA users and user roles, along with our data stores.”

Once the initial infrastructure was in place and the remaining data streams connected, Codit revised the solution’s resampling strategy to offer richer data views to end users, allowing visualization of extended vessels’ tracks.

“Every 6 seconds, the EMSA system gathers a massive amount of data points from vessels’ movements,” says Verstraete, “Logical resampling is essential to ensure that end users can see the full picture of the information that they need. To provide a richer view, we redistributed the gathered information over different partitions and restructured the data so that it’s rearranged to show the 10,000 most important positions from the requested range of time. Additionally, this sampling strategy is configurable so that it can change in line with any future increase or decrease in that limit.”

With its cloud-based solution in place, EMSA saw an opportunity to further enhance its storage for the longer term and explore big data through Microsoft Azure Databricks, a highly performant technology for running queries on a large amount of data. Codit integrated the big-data technology into the new solution, storing every piece of position data inside Azure Databricks, from where EMSA can query the data through workbooks and front-end applications can retrieve the information through APIs.