Generated from the repository · 2026-09-05
What this is built with,
and where to check.
Every row below is read out of the repository by
ops/stack — the dependency files for
what is declared, the pipeline source for which satellite products each case
actually ingests, a grep for which estimators are actually fitted. Nothing here is
typed by hand, which means nothing here can name a library the code does not
install. Where a row links to a file, that is the file that uses it.
- Data sources
- 16 external, all publicly obtainable
- Python packages
- 39 pinned per component, each pipeline and the assistant its own uv project
- Static apps
- 11 every route prerendered, no server rendering
- Served from
- 1 2-core, 1.9 GB virtual machine
Sixteen sources, sixteen different shapes
This is most of the work and the least visible part of it. Black Marble arrives as HDF5 on an authenticated NASA endpoint; ERA5 arrives through a request queue that can take hours; Sentinel-1 arrives as compressed GRD scenes that have to be found by STAC query, orbit-corrected and reduced to per-cell time series before a model sees anything. The counts below are cases whose pipeline code reads the source, not cases that cite it.
| Source | What it gives | Structure | Cases using it |
|---|---|---|---|
| OpenStreetMap | roads, land use, points of interest | Vector · nodes, ways and relations | 5 |
| geoBoundaries | administrative boundaries, open licence | Vector · administrative boundaries | 5 |
| HDX | humanitarian datasets | Mixed · tables or vector, per dataset | 4 |
| NASA Black Marble / VIIRS | calibrated night-time radiance | Raster · HDF5 tiles | 4 |
| BPS (Statistics Indonesia) | the official statistics every case is scored against | Tables · official statistics by region | 3 |
| ERA5 reanalysis | hourly weather back to 1940, via the Copernicus CDS API | Gridded time series · NetCDF | 3 |
| NASA FIRMS active fire | thermal anomaly detections, per overpass | Points · one row per detection | 3 |
| STAC catalogue search | asset discovery instead of scraping a directory listing | JSON API · asset metadata | 3 |
| Sentinel-2 optical | 10 m multispectral, for indices and land cover | Raster · multispectral tiles | 3 |
| Copernicus GLO-30 DEM | 30 m elevation | Raster · elevation | 2 |
| Sentinel-1 SAR | C-band radar backscatter, sees through monsoon cloud | Raster · radar scenes | 2 |
| Copernicus EMS / EFFIS | atmospheric composition and fire emissions | Gridded · composition and emissions | 1 |
| GDELT | global news event and tone feeds | Event records · one row per news event | 1 |
| GTFS feeds | published transit timetables | Zipped CSV tables · transit schedules | 1 |
| Google Open Buildings | building footprints across the global south | Vector · building footprints | 1 |
| UN Comtrade | bilateral trade flows by product | Tables · bilateral flows by product | 1 |
Publicly obtainable is not the same as open. Most of these need a free account and an API key, one moved behind a paid wall mid-project, and several datasets were excluded on licence — including two whose exclusion cost a case its planned independent cross-check. The full accounting is on Standards.
The geospatial and data layer
Used by two or more of the 10 cases. Each pipeline is a separate
uv project with its own pinned dependencies, so one case cannot break
another and any single case can be re-run on its own.
| Package | Version | What it does here | Cases |
|---|---|---|---|
| numpy | >=1.26, unpinned | arrays, and the per-cell radar stacks | 11 |
| pandas | >=2.2 | the tabular spine of every pipeline | 10 |
| pyarrow | >=17 | Parquet on the way in and out; every intermediate is columnar | 10 |
| requests | >=2.32 | HTTP against a dozen agency APIs, each with its own idea of an error | 10 |
| duckdb | >=1.1 | SQL over Parquet without standing up a database | 8 |
| geopandas | >=1.0 | vector geometry with a coordinate reference system attached | 7 |
| rasterio | >=1.3 | reading and windowing satellite rasters without loading the scene | 7 |
| scipy | >=1.14 | signal filtering, interpolation, distribution fitting | 7 |
| shapely | >=2.0, unpinned | geometry predicates and repair | 7 |
| exactextract | >=0.2 | area-weighted zonal statistics — the correct way to summarise a raster over a polygon, rather than counting pixels whose centre falls inside | 6 |
| pillow | >=10.4 | encodes Jakarta's velocity field as a lookup PNG, and composes its hero image | 5 |
| pyogrio | >=0.10 | the fast Arrow path into and out of vector formats | 5 |
| scikit-learn | >=1.5 | the estimators, the calibration, and permutation importance | 4 |
| anthropic | >=0.40 | declared, not yet running. shared/pipeline/brief.py turns a stats JSON into a short brief with one low-temperature call whose prompt forbids any number absent from the JSON, gated on a human merging the pull request — but nothing calls it yet, so no output on this site is model-written | 3 |
| cdsapi | >=0.7, >=0.7.7 | queued requests to the Copernicus Climate Data Store | 3 |
| netcdf4 | >=1.6, >=1.7 | the format ERA5 arrives in | 3 |
| pystac-client | >=0.8 | STAC search, so asset discovery is a query rather than a scrape | 3 |
| xarray | >=2024.6, >=2024.7 | labelled n-dimensional arrays for the climate stacks | 3 |
| boto3 | >=1.34 | S3-hosted public buckets | 2 |
| fastapi | >=0.115 | 2 | |
| lightgbm | >=4.5 | gradient boosting, through the native booster API | 2 |
| shap | >=0.46 | per-prediction attribution, exported so the dashboard can show why | 2 |
| torch | >=2.14, >=2.4 | the two learned models — a 1-D CNN over radar time series, a dilated CNN over fire-weather sequences — trained on a rented T4, not on this box | 2 |
| uvicorn | >=0.32 | 2 |
Specialist, one case each
| Package | What it does here | Case |
|---|---|---|
| ecomplexity | economic complexity indices | trade-complexity |
| google-genai | platform/knowledge | |
| h5py | HDF5, which is how NASA ships Black Marble | nightlights-pulse |
| hijridate | the Hijri calendar — Ramadan moves through the solar year and it shows up in both night-time lights and the harvest calendar | nightlights-pulse |
| networkx | the trade graph | trade-complexity |
| openpyxl | platform/knowledge | |
| pydantic | platform/assistant | |
| pymupdf | platform/knowledge | |
| pyproj | rice-security | |
| python-docx | platform/knowledge | |
| python-multipart | platform/knowledge | |
| r5py | the R5 routing engine — real multimodal travel times from a timetable, not straight-line buffers | transit-equity |
| s2sphere | S2 cells, for indexing points on a sphere | poverty-map |
| sentence-transformers | platform/knowledge | |
| statsmodels | the statistical tests | nightlights-pulse |
What is actually fitted
Gradient boosting and regularised linear models on raster-derived features. Listed with the file that fits them, because the interesting part of a model is rarely the estimator — it is the loss, the calibration, the weights and the metric.
| What | Why it is that and not something else | Case |
|---|---|---|
| LightGBM, native booster API | gradient boosting through lgb.train with explicit params, sample weights and a Dataset — not the sklearn wrapper | fire-haze, poverty-map |
| SHAP | per-prediction attribution, exported to the dashboard | fire-haze, poverty-map |
| HistGradientBoostingRegressor | histogram gradient boosting | air-quality |
| IsotonicRegression | monotone calibration of predicted probabilities | fire-haze |
| RidgeCV | L2 linear regression with the penalty chosen by cross-validation, inside a Pipeline | poverty-map |
| average_precision_score | PR-AUC, which is the honest metric on an imbalanced target | fire-haze |
| gamma distribution fit | parametric fit for a strictly positive skewed quantity | fire-haze |
| permutation_importance | importance measured by degrading the input, not by split gain | air-quality |
| quantile loss | prediction intervals rather than a point estimate | air-quality |
| roc_auc_score | ranking quality | fire-haze |
How they are scored matters more than what they are, and it is where the results actually came from. Every model is scored against a naive rival that already contains the easy knowledge, and the validation is spatially blocked rather than randomly folded. On the poverty case that single decision was the finding: a random k-fold reported R² 0.65 where a spatially-blocked design reported 0.395, and 57% of the squared error turned out to be one constant per province — the model was recovering the survey's own offsets, not seeing poverty.
The night-lights case is the other half of the same discipline: the nowcast came out 41% worse than doing nothing, with 99.5% of it intercept — a number that would print almost unchanged if the satellite were switched off. It is published in that state. How that gets decided.
Eleven static apps, no server rendering
Each case is its own Astro app with its own base path, built to static files. There is no backend behind any dashboard: the numbers are computed in the pipeline and shipped as JSON, or recomputed in the browser from a small table. Where a figure has tens of thousands of marks, the dense layer goes on a canvas so the axis text stays selectable text.
| Package | Version | What it does here | Apps |
|---|---|---|---|
| astro | ^5.1.0 | one static app per case plus this site; every route prerendered, no server rendering anywhere | 12 |
| d3 | ^7.9.0 | scales, axes, shape generators and hit-testing. The low-level pieces — the charts are composed from them rather than configured | 10 |
| maplibre-gl | ^5.0.0 | vector tile maps, self-hosted | 5 |
| three | ^0.169.0 | the 3D trade network view | 1 |
No high-level charting library — no Plotly, no Chart.js, no Vega. Charts are composed from d3's scales, axes and shape generators, which means the projection, the domain and the hit-testing are decisions in the code rather than options in a config object. Where a chart has tens of thousands of marks, the dense layer goes on a canvas and the axes, legend and labels stay in SVG so the text is still text.
Serving this is small. Rebuilding it is not.
The footprint below is what it costs to serve these pages, and it would be easy to read it as what it costs to reproduce them. It is not, by more than an order of magnitude. Two of the ten cases train a convolutional network on a GPU, and several ingest stages peak in gigabytes while reprojecting satellite granules. Anyone who cloned this and started on the serving box would meet the memory wall rather than a result — which is how this project lost three pipeline runs to out-of-memory kills, each against a hand-typed ceiling rather than a measured one.
- To serve — two cores and 1.9 GB, as described below. Nothing on the public site needs more.
- To train fire-haze’s network —
g4dn.xlarge: 4 vCPU, 15.4 GB, Tesla T4 with 14.6 GB VRAM. folds at batch 16 over a 88,179-parameter model, last run 2026-09-05. Training parameters recorded indl_results.json; the hardware stamped by the run itself. - To train rice-security’s network —
g4dn.xlarge: 4 vCPU, 16 GB, NVIDIA T4 with 16 GB VRAM. 6 folds at batch 4,096 over a 50,838-parameter model, last run 2026-09-02. Training parameters recorded indl_results.json; the hardware declared by the operator, not yet stamped by a run. - To run the ingest stages — several gigabytes each, and more disk than
anything else here: the fetched granules, reanalysis fields and API responses that
.gitignoreexcludes.
What this page will not tell you yet, and why. There is no honest number here
for peak memory. The gauge that should have recorded it read ru_maxrss
as bytes on Linux, where it is kilobytes, so the run on the machine that mattered
reported 4.22 GB — a
memory gauge failing a thousandfold low on the GPU box, the one place it was needed.
The arithmetic is fixed and
ops/pipeline now derives every
cap from a recorded peak rather than a guess, but the stored record still carries
that zero and the ledger behind it holds one row. So the figure is absent rather
than estimated, and it appears here once a run has measured it. A remembered number
in this slot would contradict the
page that promises none of these are typed.
What can be said without measuring it is the
ceiling: the training host carries 15.4 GB, so the peak
was under that or the run would have died — which is a bound, not a
measurement.
Eleven apps on two cores and 1.9 GB
Eleven Astro dev servers cost about 4.5 GB between them, which would have needed a four-times-larger machine to serve pages that are already fully static. So one nginx serves the built output of all of them from a single origin, each case under its own slug.
ops/serve-staticderives port, base path and directory for every app from the repository, then generates the entire nginx configuration — server blocks, redirects for retired paths, aliases for paths a reader would guess, cache headers for hashed assets. The config is not hand-edited, so it cannot disagree with the apps it serves.- HTTPS via Let's Encrypt on a multi-name certificate, renewed automatically.
- Deploys are an rsync and a build, one app at a time, because two concurrent Astro builds will exhaust the memory on this box.
ops/reviewruns house rules against the rendered output rather than the source, so a check cannot pass on a template that fails in the page.ops/glossarykeeps one set of plain-language definitions in step across ten separate Astro apps, and each article scans its own source at build time to render only the terms it uses — no per-article list to go stale.
Why this is on the page. Anyone can produce a notebook. The part that takes judgement is the part that keeps working: pinned per-case environments, generated configuration, checks that run against the rendered artifact, and a deployment that fits on the hardware you actually have. The whole repository is public.
Found something wrong on this page? Report a correction — it opens a pre-filled issue — or email taufik.adi@openstudy.id. Corrections are credited by name in the errata, and one that changes a finding says so on the page.