ANM Data API · Alcides Oliveira

The problem

ÍGNEA’s databases and map tools are built on public data from Brazil’s National Mining Agency (ANM). The agency publishes it on its open data portal as folders of CSV, ZIP, shapefile, and ODS files, refreshes them between 06:00 and 06:30, and can take minutes to start sending a large file. In 2026 the portal also moved to a new domain, and the old address had a shutdown date. ÍGNEA needed one service that downloads each dataset once, keeps a dated copy, and tells the right database when a new version lands.

Decisions

One registry, three handlers

I described every dataset as data and wrote one handler per format (ZIP, CSV folder, shapefile) instead of one script per dataset. New datasets became a config change, at the cost of forcing each dataset’s quirks into a few fields such as allowlists and blocked extensions.

Push after upload, cron as a fallback

The API notifies each database service right after its files reach S3, so the databases load new data without polling. The database answers at once and loads in the background, a failed trigger gets one retry and never blocks the wave, and each database service keeps its own daily scheduled sync, so a missed trigger costs a delay and not a lost update.

Moving map layers out of the API

I first converted the SIGMINE shapefiles to GeoJSON inside this service with proj4 and served static files from S3. That broke the pattern the other datasets followed, so I moved the eight layers into their own PostGIS service and kept the API as a proxy with the same URLs, so the map clients did not have to change.

What I built

A config-driven dataset registry in TypeScript. Adding a dataset means one entry with its portal path, format, file allowlist or blocked extensions, S3 prefix, and schedule hour, plus a line in the trigger map when a database consumes it.

Streaming ZIP extraction from S3 back to S3 with unzipper and the AWS SDK’s lib-storage, with a stall timer that aborts an upload when it stops making progress.

A manifest check for the SCM microdata package, added after a run stalled mid-extraction and published 10 of 28 files. The sync now fails unless all 28 expected text files are present, the databases get no trigger from a failed or skipped run, and an environment switch can turn the check off for an emergency rollback.

A trigger service in Axios: one POST per downstream database with a 15-second timeout, one retry after 30 seconds, a log line per attempt, and a success count per wave. It never throws, so one service that is down cannot stop the rest of the wave.

An ODS parser (adm-zip plus reading the sheet’s XML) that opens ANM’s metadata spreadsheets and publishes each dataset’s field descriptions as JSON in S3. It runs after the sync with its errors caught, so a broken sheet cannot fail a download.

The Express REST API: Helmet, a limit of 100 requests per minute, API keys for downloads and manual triggers, and a one-hour NodeCache on portal listings. Terraform holds the S3 lifecycle rules that move files to Glacier after 90 days and to Deep Archive after a year.

How it works

A TypeScript registry describes 15 datasets: the portal folder, the format (a ZIP package, a folder of CSV files, or shapefile ZIPs), which files to keep, the S3 prefix, and the hour it runs. node-cron groups them into waves at 07:00, 08:00, 09:00, 10:00, and 11:00 Brasília time, after ANM’s morning update. Each job reads the portal’s directory listing, compares the Last-Modified dates against the dataset’s sync history, and skips anything that has not changed.

Downloads stream from the portal into S3 without touching the disk. The SCM ZIP package is stored as it came, then read back from S3 and extracted file by file through multipart uploads. Everything lands under dated prefixes, and a sync-history JSON per dataset keeps the last 30 runs. After a sync, a separate step turns ANM’s ODS metadata sheets into JSON with each field’s name, type, and description.

When a dataset finishes, the scheduler sends a POST to each database service that consumes it. There are 14 of these services, and each loads the files from S3 into PostgreSQL. The SCM package feeds two of them, and each of the eight SIGMINE map layers sends its own layer ID. The same Express app serves the stored files through a REST API and forwards SIGMINE GeoJSON requests to the PostGIS service.

Built with

Back end

TypeScript, Node.js, Express, node-cron, Axios, unzipper, adm-zip, csv-parser, NodeCache

Security

Helmet, express-rate-limit, CORS, API keys

Storage

AWS S3 (SDK v3, lib-storage multipart uploads), S3 Glacier and Deep Archive lifecycle rules

Infrastructure

AWS Lightsail, PM2, Terraform

Downstream services

PostgreSQL, PostGIS, Fastify