openstudy.id

Generated from the repository · 2026-09-05

What this is built with,
and where to check.

Every row below is read out of the repository by ops/stack — the dependency files for what is declared, the pipeline source for which satellite products each case actually ingests, a grep for which estimators are actually fitted. Nothing here is typed by hand, which means nothing here can name a library the code does not install. Where a row links to a file, that is the file that uses it.

Data sources
16
external, all publicly obtainable
Python packages
39
pinned per component, each pipeline and the assistant its own uv project
Static apps
11
every route prerendered, no server rendering
Served from
1
2-core, 1.9 GB virtual machine
Datawhere it comes from

Sixteen sources, sixteen different shapes

This is most of the work and the least visible part of it. Black Marble arrives as HDF5 on an authenticated NASA endpoint; ERA5 arrives through a request queue that can take hours; Sentinel-1 arrives as compressed GRD scenes that have to be found by STAC query, orbit-corrected and reduced to per-cell time series before a model sees anything. The counts below are cases whose pipeline code reads the source, not cases that cite it.

SourceWhat it givesStructure Cases using it
OpenStreetMap roads, land use, points of interest Vector · nodes, ways and relations 5
geoBoundaries administrative boundaries, open licence Vector · administrative boundaries 5
HDX humanitarian datasets Mixed · tables or vector, per dataset 4
NASA Black Marble / VIIRS calibrated night-time radiance Raster · HDF5 tiles 4
BPS (Statistics Indonesia) the official statistics every case is scored against Tables · official statistics by region 3
ERA5 reanalysis hourly weather back to 1940, via the Copernicus CDS API Gridded time series · NetCDF 3
NASA FIRMS active fire thermal anomaly detections, per overpass Points · one row per detection 3
STAC catalogue search asset discovery instead of scraping a directory listing JSON API · asset metadata 3
Sentinel-2 optical 10 m multispectral, for indices and land cover Raster · multispectral tiles 3
Copernicus GLO-30 DEM 30 m elevation Raster · elevation 2
Sentinel-1 SAR C-band radar backscatter, sees through monsoon cloud Raster · radar scenes 2
Copernicus EMS / EFFIS atmospheric composition and fire emissions Gridded · composition and emissions 1
GDELT global news event and tone feeds Event records · one row per news event 1
GTFS feeds published transit timetables Zipped CSV tables · transit schedules 1
Google Open Buildings building footprints across the global south Vector · building footprints 1
UN Comtrade bilateral trade flows by product Tables · bilateral flows by product 1

Publicly obtainable is not the same as open. Most of these need a free account and an API key, one moved behind a paid wall mid-project, and several datasets were excluded on licence — including two whose exclusion cost a case its planned independent cross-check. The full accounting is on Standards.

Pipelines10 uv projects

The geospatial and data layer

Used by two or more of the 10 cases. Each pipeline is a separate uv project with its own pinned dependencies, so one case cannot break another and any single case can be re-run on its own.

PackageVersionWhat it does hereCases
numpy >=1.26, unpinned arrays, and the per-cell radar stacks 11
pandas >=2.2 the tabular spine of every pipeline 10
pyarrow >=17 Parquet on the way in and out; every intermediate is columnar 10
requests >=2.32 HTTP against a dozen agency APIs, each with its own idea of an error 10
duckdb >=1.1 SQL over Parquet without standing up a database 8
geopandas >=1.0 vector geometry with a coordinate reference system attached 7
rasterio >=1.3 reading and windowing satellite rasters without loading the scene 7
scipy >=1.14 signal filtering, interpolation, distribution fitting 7
shapely >=2.0, unpinned geometry predicates and repair 7
exactextract >=0.2 area-weighted zonal statistics — the correct way to summarise a raster over a polygon, rather than counting pixels whose centre falls inside 6
pillow >=10.4 encodes Jakarta's velocity field as a lookup PNG, and composes its hero image 5
pyogrio >=0.10 the fast Arrow path into and out of vector formats 5
scikit-learn >=1.5 the estimators, the calibration, and permutation importance 4
anthropic >=0.40 declared, not yet running. shared/pipeline/brief.py turns a stats JSON into a short brief with one low-temperature call whose prompt forbids any number absent from the JSON, gated on a human merging the pull request — but nothing calls it yet, so no output on this site is model-written 3
cdsapi >=0.7, >=0.7.7 queued requests to the Copernicus Climate Data Store 3
netcdf4 >=1.6, >=1.7 the format ERA5 arrives in 3
pystac-client >=0.8 STAC search, so asset discovery is a query rather than a scrape 3
xarray >=2024.6, >=2024.7 labelled n-dimensional arrays for the climate stacks 3
boto3 >=1.34 S3-hosted public buckets 2
fastapi >=0.115 2
lightgbm >=4.5 gradient boosting, through the native booster API 2
shap >=0.46 per-prediction attribution, exported so the dashboard can show why 2
torch >=2.14, >=2.4 the two learned models — a 1-D CNN over radar time series, a dilated CNN over fire-weather sequences — trained on a rented T4, not on this box 2
uvicorn >=0.32 2

Specialist, one case each

PackageWhat it does hereCase
ecomplexity economic complexity indices trade-complexity
google-genai platform/knowledge
h5py HDF5, which is how NASA ships Black Marble nightlights-pulse
hijridate the Hijri calendar — Ramadan moves through the solar year and it shows up in both night-time lights and the harvest calendar nightlights-pulse
networkx the trade graph trade-complexity
openpyxl platform/knowledge
pydantic platform/assistant
pymupdf platform/knowledge
pyproj rice-security
python-docx platform/knowledge
python-multipart platform/knowledge
r5py the R5 routing engine — real multimodal travel times from a timetable, not straight-line buffers transit-equity
s2sphere S2 cells, for indexing points on a sphere poverty-map
sentence-transformers platform/knowledge
statsmodels the statistical tests nightlights-pulse
Modelsand how they are judged

What is actually fitted

Gradient boosting and regularised linear models on raster-derived features. Listed with the file that fits them, because the interesting part of a model is rarely the estimator — it is the loss, the calibration, the weights and the metric.

WhatWhy it is that and not something elseCase
LightGBM, native booster API gradient boosting through lgb.train with explicit params, sample weights and a Dataset — not the sklearn wrapper fire-haze, poverty-map
SHAP per-prediction attribution, exported to the dashboard fire-haze, poverty-map
HistGradientBoostingRegressor histogram gradient boosting air-quality
IsotonicRegression monotone calibration of predicted probabilities fire-haze
RidgeCV L2 linear regression with the penalty chosen by cross-validation, inside a Pipeline poverty-map
average_precision_score PR-AUC, which is the honest metric on an imbalanced target fire-haze
gamma distribution fit parametric fit for a strictly positive skewed quantity fire-haze
permutation_importance importance measured by degrading the input, not by split gain air-quality
quantile loss prediction intervals rather than a point estimate air-quality
roc_auc_score ranking quality fire-haze

How they are scored matters more than what they are, and it is where the results actually came from. Every model is scored against a naive rival that already contains the easy knowledge, and the validation is spatially blocked rather than randomly folded. On the poverty case that single decision was the finding: a random k-fold reported R² 0.65 where a spatially-blocked design reported 0.395, and 57% of the squared error turned out to be one constant per province — the model was recovering the survey's own offsets, not seeing poverty.

The night-lights case is the other half of the same discipline: the nowcast came out 41% worse than doing nothing, with 99.5% of it intercept — a number that would print almost unchanged if the satellite were switched off. It is published in that state. How that gets decided.

Interfaces11 static apps

Eleven static apps, no server rendering

Each case is its own Astro app with its own base path, built to static files. There is no backend behind any dashboard: the numbers are computed in the pipeline and shipped as JSON, or recomputed in the browser from a small table. Where a figure has tens of thousands of marks, the dense layer goes on a canvas so the axis text stays selectable text.

PackageVersionWhat it does hereApps
astro ^5.1.0 one static app per case plus this site; every route prerendered, no server rendering anywhere 12
d3 ^7.9.0 scales, axes, shape generators and hit-testing. The low-level pieces — the charts are composed from them rather than configured 10
maplibre-gl ^5.0.0 vector tile maps, self-hosted 5
three ^0.169.0 the 3D trade network view 1

No high-level charting library — no Plotly, no Chart.js, no Vega. Charts are composed from d3's scales, axes and shape generators, which means the projection, the domain and the hit-testing are decisions in the code rather than options in a config object. Where a chart has tens of thousands of marks, the dense layer goes on a canvas and the axes, legend and labels stay in SVG so the text is still text.

Reproducingnot the same box

Serving this is small. Rebuilding it is not.

The footprint below is what it costs to serve these pages, and it would be easy to read it as what it costs to reproduce them. It is not, by more than an order of magnitude. Two of the ten cases train a convolutional network on a GPU, and several ingest stages peak in gigabytes while reprojecting satellite granules. Anyone who cloned this and started on the serving box would meet the memory wall rather than a result — which is how this project lost three pipeline runs to out-of-memory kills, each against a hand-typed ceiling rather than a measured one.

  • To serve — two cores and 1.9 GB, as described below. Nothing on the public site needs more.
  • To train fire-haze’s networkg4dn.xlarge: 4 vCPU, 15.4 GB, Tesla T4 with 14.6 GB VRAM. folds at batch 16 over a 88,179-parameter model, last run 2026-09-05. Training parameters recorded in dl_results.json; the hardware stamped by the run itself.
  • To train rice-security’s networkg4dn.xlarge: 4 vCPU, 16 GB, NVIDIA T4 with 16 GB VRAM. 6 folds at batch 4,096 over a 50,838-parameter model, last run 2026-09-02. Training parameters recorded in dl_results.json; the hardware declared by the operator, not yet stamped by a run.
  • To run the ingest stages — several gigabytes each, and more disk than anything else here: the fetched granules, reanalysis fields and API responses that .gitignore excludes.

What this page will not tell you yet, and why. There is no honest number here for peak memory. The gauge that should have recorded it read ru_maxrss as bytes on Linux, where it is kilobytes, so the run on the machine that mattered reported 4.22 GB — a memory gauge failing a thousandfold low on the GPU box, the one place it was needed. The arithmetic is fixed and ops/pipeline now derives every cap from a recorded peak rather than a guess, but the stored record still carries that zero and the ledger behind it holds one row. So the figure is absent rather than estimated, and it appears here once a run has measured it. A remembered number in this slot would contradict the page that promises none of these are typed. What can be said without measuring it is the ceiling: the training host carries 15.4 GB, so the peak was under that or the run would have died — which is a bound, not a measurement.

Servingone small box

Eleven apps on two cores and 1.9 GB

Eleven Astro dev servers cost about 4.5 GB between them, which would have needed a four-times-larger machine to serve pages that are already fully static. So one nginx serves the built output of all of them from a single origin, each case under its own slug.

  • ops/serve-static derives port, base path and directory for every app from the repository, then generates the entire nginx configuration — server blocks, redirects for retired paths, aliases for paths a reader would guess, cache headers for hashed assets. The config is not hand-edited, so it cannot disagree with the apps it serves.
  • HTTPS via Let's Encrypt on a multi-name certificate, renewed automatically.
  • Deploys are an rsync and a build, one app at a time, because two concurrent Astro builds will exhaust the memory on this box.
  • ops/review runs house rules against the rendered output rather than the source, so a check cannot pass on a template that fails in the page.
  • ops/glossary keeps one set of plain-language definitions in step across ten separate Astro apps, and each article scans its own source at build time to render only the terms it uses — no per-article list to go stale.

Why this is on the page. Anyone can produce a notebook. The part that takes judgement is the part that keeps working: pinned per-case environments, generated configuration, checks that run against the rendered artifact, and a deployment that fits on the hardware you actually have. The whole repository is public.

Found something wrong on this page? Report a correction — it opens a pre-filled issue — or email taufik.adi@openstudy.id. Corrections are credited by name in the errata, and one that changes a finding says so on the page.