nprasad2077/NBA_Go
1
0
Fork 0
mirror of https://github.com/nprasad2077/NBA_Go.git synced 2026-09-22 22:15:13 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: update README.md with live endpoints, dual-stack architecture, and latest usage instructions

Browse Source
This commit is contained in:
nprasad2077
2026-09-09 15:36:10 -05:00
parent a4ac5fcff5
commit 53e47ac967
1 changed files with 215 additions and 161 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+215 -161
View File
@@ -1,246 +1,313 @@
# NBA_Go # NBA_Go
A high-performance NBA statistics REST API built with Go (Fiber), PostgreSQL, and NGINX. Data is scraped from Basketball Reference and served through a load-balanced, containerized stack with built-in observability. A high-performance NBA statistics REST API built with Go (Fiber), PostgreSQL, and NGINX. Data is scraped from Basketball Reference and served through a load-balanced, containerized stack with dual-stack IPv4/IPv6 support, read/write database splitting, and built-in Prometheus & Grafana observability.
---
## Live Production Endpoints
| Service / Endpoint | URL | Description |
| :--- | :--- | :--- |
| **API Base URL** | `https://nba.turbo-data.com` | Production API root (redirects to Swagger UI) |
| **Swagger UI Docs** | `https://nba.turbo-data.com/swagger/index.html` | Interactive API documentation & sandbox |
| **Player Totals** | `https://nba.turbo-data.com/api/playertotals` | Season totals with pagination & sorting |
| **Games & Box Scores** | `https://nba.turbo-data.com/api/games` | Game schedules, box scores, line scores |
| **Advanced Stats** | `https://nba.turbo-data.com/api/playeradvancedstats` | Advanced metrics (PER, WS, VORP, BPM) |
| **Shot Charts** | `https://nba.turbo-data.com/api/playershotchart` | Shot coordinate & outcome data |
| **Liveness Check** | `https://nba.turbo-data.com/health/live` | Process liveness probe |
| **Readiness Check** | `https://nba.turbo-data.com/health/ready` | Database readiness probe |
| **Metrics** | `https://nba.turbo-data.com/metrics` | Prometheus metrics endpoint |
| **Coolify Dashboard** | `https://turbo-data.com` | Infrastructure & deployment management |
---
## Architecture ## Architecture
``` ```
┌──────────────────────────────────────────────────────────┐ ┌──────────────────────────────────────────────────────────────┐
│ NGINX (reverse proxy / round-robin load balancer :8080) │ │ Traefik (Coolify Ingress Proxy :80 / :443 SSL) │
├──────────────────────────────────────────────────────────┤ └──────────────────────────────┬───────────────────────────────┘
│ API Instance x3 (Fiber :5000 each) │ │
│ ┌──────────┐ ┌─────────────┐ ┌───────────────────┐ │ ▼
│ │ Routes │→ │ Controllers │→ │ Services (scraper) │ │ ┌──────────────────────────────────────────────────────────────┐
│ └──────────┘ └─────────────┘ └───────────────────┘ │ │ NGINX (Reverse Proxy, API Cache, Round-Robin Load Balancer) │
├──────────────────────────────────────────────────────────┤ │ - Container port 8080 (Mapped to Host :8081) │
│ PostgreSQL 15 (GORM ORM) │ │ - Dual-stack IPv4/IPv6 upstream resolution │
├──────────────────────────────────────────────────────────┤ │ - 30s response caching for /api/* with stale-while-revalidate│
│ Prometheus + Grafana (metrics & dashboards) │ ├──────────────────────────────────────────────────────────────┤
└──────────────────────────────────────────────────────────┘ │ API Instances x3 (Fiber on [::]:5000 Dual-Stack) │
│ ┌──────────┐ ┌─────────────┐ ┌───────────────────────┐ │
│ │ Routes │→ │ Controllers │→ │ GORM DBResolver (R/W) │ │
│ └──────────┘ └─────────────┘ └───────────┬───────────┘ │
├─────────────────────────────────────────────┼────────────────┤
│ Database Layer (PostgreSQL Cluster) │ │
│ ┌──────────────────────────────────────────┴─────────────┐ │
│ │ HAProxy Write Ingress (:5437) ➔ Primary DB (:5434) │ │
│ │ HAProxy Read Ingress (:5438) ➔ Read Replicas (:5435/6) │ │
│ └────────────────────────────────────────────────────────┘ │
├──────────────────────────────────────────────────────────────┤
│ Observability │
│ - Prometheus (Scrapes /metrics on each instance, Host :9091)│
│ - Grafana (Pre-provisioned dashboards, Host :3001) │
└──────────────────────────────────────────────────────────────┘
``` ```
### Key Architectural Highlights:
1. **Dual-Stack Networking**: The Go Fiber backend listens on `[::]:5000` (`net.Listen("tcp", ":5000")` with `app.Listener`), accepting incoming connections seamlessly over both IPv6 and IPv4 within Docker and Coolify networks.
2. **Read/Write DB Splitting**: GORM uses the `dbresolver` plugin to automatically route all write operations to the PostgreSQL Primary via HAProxy port 5437, while load balancing read queries across Read Replicas via HAProxy port 5438.
3. **Multi-Layer Caching & Rate Limiting**: NGINX provides an in-memory cache (`api_cache`) for 30 seconds, while Fiber middleware enforces per-client rate limiting (20 req/min per instance, ~60 req/min effective across 3 replicas).
---
### Project Structure ### Project Structure
``` ```
. .
├── main.go # Entry point (API server or import-data mode) ├── main.go # Application entry point (API server or import-data mode)
├── import.go # Bulk data import orchestration ├── import.go # Data import orchestration
├── config/ # Database initialization ├── config/
│ └── database.go # Database connection & DBResolver R/W setup
├── models/ # GORM models (Game, PlayerAdvancedStat, PlayerTotalStat, etc.) ├── models/ # GORM models (Game, PlayerAdvancedStat, PlayerTotalStat, etc.)
├── controllers/ # HTTP handlers, DTOs, pagination, filtering, sorting ├── controllers/ # HTTP handlers, DTOs, health checks, pagination
│ ├── game_controller.go
│ ├── health_controller.go # /health/live and /health/ready handlers
│ ├── player_advanced_controller.go
│ ├── player_shot_chart_controller.go
│ └── player_total_controller.go
├── routes/ # Route registration grouped by domain ├── routes/ # Route registration grouped by domain
├── services/ # Web scrapers (Basketball Reference via goquery) ├── services/ # Scrapers (Basketball Reference via goquery)
├── utils/ ├── utils/
│ ├── middleware/ # Rate limiter, metrics, API key auth │ ├── middleware/ # Rate limiter, Prometheus metrics, API key auth
│ ├── metrics/ # Prometheus counter/histogram definitions │ ├── metrics/ # Prometheus counter & histogram definitions
│ └── security/ # API key generation & hashing │ └── security/ # API key hashing & generation
├── nginx/ # NGINX load balancer config ├── nginx/ # NGINX reverse proxy & cache configuration
├── prometheus/ # Prometheus scrape config ├── prometheus/ # Prometheus scrape configuration
├── grafana/ # Pre-provisioned dashboards & datasources ├── grafana/ # Pre-provisioned dashboards & datasources
├── docker-compose.yml # Production (Coolify) ├── docker-compose.yml # Production deployment configuration (Coolify)
├── docker-compose.local.yml # Local development (includes Postgres) ├── docker-compose.local.yml # Local development configuration (includes local Postgres)
└── docker-compose.override.yml # Override for remote DB development └── docker-compose.override.yml # Override for remote DB development
``` ```
---
## API Endpoints ## API Endpoints
| Method | Path | Description | | Method | Path | Description | Public |
| ------ | -------------------------- | --------------------------------------------------------- | | :--- | :--- | :--- | :---: |
| GET | `/api/games` | Game data with box scores, line scores, team/player stats | | `GET` | `/` | Redirects to `/swagger/index.html` | Yes |
| GET | `/api/playeradvancedstats` | Advanced stats (PER, WS, VORP, BPM, etc.) | | `GET` | `/swagger/*` | Interactive Swagger UI documentation | Yes |
| GET | `/api/playertotals` | Season totals (points, rebounds, assists, etc.) | | `GET` | `/api/games` | Paginated games with box scores, line scores, team/player stats | Yes |
| GET | `/api/playershotchart` | Shot chart coordinate data | | `GET` | `/api/playeradvancedstats` | Advanced metrics (PER, WS, VORP, BPM, etc.) | Yes |
| GET | `/swagger/*` | Interactive Swagger UI documentation | | `GET` | `/api/playertotals` | Season totals (points, rebounds, assists, etc.) | Yes |
| GET | `/metrics` | Prometheus metrics endpoint | | `GET` | `/api/playershotchart` | Shot chart coordinates and made/missed outcomes | Yes |
| POST | `/admin/keys` | Create API key (requires `X-Admin-Secret` header) | | `GET` | `/health/live` | Process liveness probe | Yes |
| `GET` | `/health/ready` | Database connection readiness probe | Yes |
| `GET` | `/metrics` | Prometheus metrics scrape endpoint | Yes |
| `POST`| `/admin/keys` | Create API key (requires `X-Admin-Secret` header) | Admin |
| `POST`| `/admin/keys/:id/revoke` | Revoke API key (requires `X-Admin-Secret` header) | Admin |
### Query Parameters (all data endpoints) ---
| Parameter | Type | Description | ### Query Parameters (Data Endpoints)
| ----------- | ------ | -------------------------------------------- |
| `page` | int | Page number (default: 1) |
| `pageSize` | int | Results per page (default: 20) |
| `sortBy` | string | Field to sort by (varies per endpoint) |
| `ascending` | bool | Sort direction (default: false / descending) |
| `season` | int | Filter by season year (e.g., 2025) |
| `team` | string | Filter by team abbreviation (e.g., LAL, BOS) |
| `playerId` | string | Filter by player ID (e.g., jamesle01) |
| `isPlayoff` | bool | Filter for playoff stats |
#### Games-specific parameters | Parameter | Type | Default | Description |
| :--- | :--- | :--- | :--- |
| `page` | `int` | `1` | Page number |
| `pageSize` | `int` | `20` | Results per page (max 100) |
| `sortBy` | `string` | varies | Field to sort by (e.g., `points`, `date`, `winShares`) |
| `ascending` | `bool` | `false` | Sort ascending (`true`) or descending (`false`) |
| `season` | `int` | - | Filter by season year (e.g., `2024`, `2025`) |
| `team` | `string` | - | Filter by team abbreviation (e.g., `LAL`, `BOS`, `GSW`) |
| `playerId` | `string` | - | Filter by player ID (e.g., `jamesle01`, `curryst01`) |
| `isPlayoff` | `bool` | `false` | Filter for playoff games or stats |
| Parameter | Type | Description | #### Additional Parameters for `/api/games`:
| --------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------- | - `date`: Filter by exact date (`YYYY-MM-DD`).
| `date` | string | Filter by date (YYYY-MM-DD) | - `gameId`: Filter by specific game ID (e.g., `202501010LAL`).
| `gameId` | string | Filter by specific game ID | - `include`: Comma-separated associations to preload (`lineScores`, `playerGameBasicStats`, `playerGameAdvStats`, `teamGameBasicStats`, `teamGameAdvStats`).
| `include` | string | Comma-separated associations to preload: `lineScores`, `playerGameBasicStats`, `playerGameAdvStats`, `teamGameBasicStats`, `teamGameAdvStats` |
---
### Example Requests ### Example Requests
#### Production (Live API)
```bash ```bash
# Get top scorers for the 2025 season # Get top scorers for the 2025 season
curl "http://localhost:8080/api/playertotals?season=2025&sortBy=points&pageSize=10" curl "https://nba.turbo-data.com/api/playertotals?season=2025&sortBy=points&pageSize=10"
# Get a specific game with full box score # Get a specific game with full box score preloaded
curl "http://localhost:8080/api/games?gameId=202501010LAL&include=lineScores,playerGameBasicStats,teamGameBasicStats" curl "https://nba.turbo-data.com/api/games?gameId=202501010LAL&include=lineScores,playerGameBasicStats,teamGameBasicStats"
# Get LeBron's advanced stats across all seasons # Get LeBron James' advanced stats across career
curl "http://localhost:8080/api/playeradvancedstats?playerId=jamesle01&sortBy=season&ascending=true" curl "https://nba.turbo-data.com/api/playeradvancedstats?playerId=jamesle01&sortBy=season&ascending=true"
# Get shot chart data for Curry in 2024 # Get Stephen Curry's shot chart data for 2024
curl "http://localhost:8080/api/playershotchart?playerId=curryst01&season=2024" curl "https://nba.turbo-data.com/api/playershotchart?playerId=curryst01&season=2024"
# Check API readiness
curl "https://nba.turbo-data.com/health/ready"
``` ```
### Response Format #### Local Development
```bash
# Query local NGINX load balancer (:8081)
curl "http://localhost:8081/api/playertotals?season=2025&pageSize=5"
All endpoints return paginated JSON: # Query local Prometheus metrics
curl "http://localhost:8081/metrics"
```
---
## Response Format
All data endpoints return structured JSON with pagination metadata:
```json ```json
{ {
"data": [...], "data": [
{
"playerId": "curryst01",
"playerName": "Stephen Curry",
"season": 2024,
"team": "GSW",
"points": 1956,
"assists": 379,
"rebounds": 330,
"threeP": 357,
"fieldPercent": 0.450
}
],
"pagination": { "pagination": {
"total": 450, "total": 450,
"page": 1, "page": 1,
"pageSize": 20, "pageSize": 20,
"pages": 22 "pages": 23
} }
} }
``` ```
## Rate Limiting ---
The API enforces a per-IP rate limit of **20 requests per minute per instance**. With 3 instances behind NGINX round-robin, the effective limit is ~60 requests/minute per client. ## Rate Limiting & Caching
Exceeding the limit returns: - **Rate Limiting**: Configured per client IP at **20 requests per minute per instance** (~60 req/min across the 3-instance cluster). If exceeded, the API responds with `HTTP 429 Too Many Requests`.
- **Response Caching**: NGINX provides an in-memory cache (`api_cache`) for **30 seconds** (`keys_zone=api_cache:10m`). Cached responses include the header `X-Cache-Status: HIT` (or `MISS`).
```json ---
HTTP 429
{"error": "Rate limit exceeded. Try again later."}
```
## Getting Started ## Getting Started
### Prerequisites ### Prerequisites
- Docker & Docker Compose - Docker & Docker Compose
- Go 1.23+ (for local development) - Go 1.23+ (for local development)
- A `.env` file with database credentials
### Environment Variables ### Environment Variables
Create a `.env` file in the project root:
```env ```env
DB_HOST=postgres DB_HOST=178.105.149.129
DB_USER=your_user DB_USER=your_user
DB_PASSWORD=your_password DB_PASSWORD=your_password
DB_NAME=your_db DB_NAME=appdb
DB_PORT=5432 DB_PORT=5432
ADMIN_SECRET=your_admin_secret ADMIN_SECRET=your_admin_secret
``` ```
### Local Development ### Running Locally with Docker
```bash ```bash
# Start everything (Postgres, 3 API instances, NGINX, Prometheus, Grafana) # Start full local stack (Postgres, 3 API replicas, NGINX, Prometheus, Grafana)
docker-compose -f docker-compose.local.yml up --build -d docker compose -f docker-compose.local.yml up --build -d
# Or use the Makefile shortcut # Or using Makefile
make up make up
``` ```
Services will be available at: Local service ports:
- **API (via NGINX)**: [http://localhost:8081](http://localhost:8081)
- **Swagger Docs**: [http://localhost:8081/swagger/index.html](http://localhost:8081/swagger/index.html)
- **Prometheus**: [http://localhost:9091](http://localhost:9091)
- **Grafana**: [http://localhost:3001](http://localhost:3001) (`admin` / `testing`)
- **API Direct Instances**: `http://localhost:5001`, `5002`, `5003`
| Service | URL | ### Initial Data Import
| ----------------------- | --------------------------------------- |
| API (via NGINX) | <http://localhost:8081> |
| Prometheus | <http://localhost:9090> |
| Grafana | <http://localhost:3001> (admin/testing) |
| API instance 1 (direct) | <http://localhost:5001> |
| API instance 2 (direct) | <http://localhost:5002> |
| API instance 3 (direct) | <http://localhost:5003> |
### Importing Data To run the initial schema migrations and scrape Basketball Reference:
The application has a dual-mode entry point. To run the initial data import (migrations + scraping):
```bash ```bash
docker-compose -f docker-compose.local.yml run --rm db-init docker compose -f docker-compose.local.yml run --rm db-init
``` ```
This runs `main.go` with the `import-data` argument, which: This launches the container in one-off `import-data` mode:
1. Executes GORM `AutoMigrate` against the Primary database.
2. Scrapes player advanced stats, regular season totals, playoff totals, and shot charts.
3. Upserts all records into PostgreSQL.
1. Runs all GORM AutoMigrate operations ### Stopping Services
2. Scrapes Basketball Reference for player advanced stats, totals, game schedules, and box scores
3. Upserts all data into PostgreSQL
### Stopping
```bash ```bash
docker compose down docker compose -f docker-compose.local.yml down
# or # or
make down make down
``` ```
## Production Deployment ---
The main `docker-compose.yml` is configured for deployment on Coolify with an external `coolify` network. It expects the database to be provisioned separately (no local Postgres service). ## Production Deployment (Coolify)
The `docker-compose.override.yml` disables the local Postgres container and removes `depends_on` constraints, allowing API services to connect to a remote database specified in `.env`. The application is deployed via Coolify on branch `shooting`:
- Uses Traefik as the edge reverse proxy with automated Let's Encrypt SSL.
- Bridges into the `coolify-shared` network to communicate with external databases and proxy services.
- Configured with multi-container services (`api1`, `api2`, `api3`, `nginx`, `prometheus`, `grafana`, `db-init`).
To trigger a redeployment from the command line on the server:
```bash
docker compose -f /data/coolify/applications/<app_uuid>/docker-compose.yaml up -d --build
```
---
## Observability ## Observability
### Prometheus Metrics ### Prometheus Metrics
Exposed at `/metrics`:
- `nba_http_requests_total` — Counter partitioned by `method`, `endpoint`, and `status`.
- `nba_http_request_duration_seconds` — Histogram tracking request latencies.
- `nba_db_operations_total` — Counter tracking database read/write queries.
Exposed at `/metrics` on each API instance. Tracked metrics: ### Grafana Dashboards
Pre-provisioned dashboards in `grafana/dashboards` visualize:
- Request throughput and error rates per endpoint.
- NGINX cache hit ratio.
- Latency percentiles (p50, p95, p99).
- Database read/write distribution across Primary and Replicas.
- `nba_http_requests_total` — counter by method, endpoint, status ---
- `nba_http_request_duration_seconds` — histogram by method, endpoint
- `nba_db_operations_total` — counter by operation, entity
### Grafana ## Swagger API Documentation
Pre-provisioned dashboards visualize request rates and endpoint usage. Access at port 3001 (local) or 3000 (production). To regenerate Swagger documentation after modifying controllers or annotations:
## API Key Management (Optional)
API key authentication is available but currently disabled. To create keys for future use:
```bash
# Create a key
curl -XPOST http://localhost:8080/admin/keys \
-H "X-Admin-Secret: $ADMIN_SECRET" \
-d '{"label":"my-app"}'
# → {"id":1, "apiKey":"ab12cd…"}
# Revoke a key
curl -XPOST http://localhost:8080/admin/keys/1/revoke \
-H "X-Admin-Secret: $ADMIN_SECRET"
```
To enforce API keys, uncomment `app.Use(middleware.APIKeyAuth(db))` in `main.go`.
## Regenerating Swagger Docs
```bash ```bash
swag init -g main.go -o docs swag init -g main.go -o docs
``` ```
## Running Tests ---
## Testing & Load Testing
### Unit & Integration Tests
```bash ```bash
go test -v . go test -v ./...
``` ```
### Load Testing ### Load Testing
The built-in load test utility supports weighted page distributions:
The load-test utility sends concurrent GET requests and can distribute them
across pages with `-pageMix`. Each entry uses the format
`page-or-range:weight`; range weights are distributed evenly across the pages
in that range. Weights are normalized automatically and do not need to total
100.
```bash ```bash
go run ./test/loadtest.go \ go run ./test/loadtest.go \
-url "http://localhost:8081/api/playertotals?page=1&pageSize=50" \ -url "https://nba.turbo-data.com/api/playertotals?page=1&pageSize=50" \
-n 500 \ -n 500 \
-c 20 \ -c 20 \
-pageMix "1-3:60,4-10:30,11-20:10" \ -pageMix "1-3:60,4-10:30,11-20:10" \
@@ -248,33 +315,20 @@ go run ./test/loadtest.go \
-log ./test/results.log -log ./test/results.log
``` ```
In this example, approximately 60% of requests target pages 1 through 3, ---
30% target pages 4 through 10, and 10% target pages 11 through 20. The
`-seed` flag makes page selection reproducible; omit it to use a time-based
seed. Without `-pageMix`, the URL is sent unchanged as in the original
load-test behavior.
The utility prints aggregate results and per-page counts, failures, and
average successful response time. Invalid page ranges, non-positive weights,
and overlapping ranges are rejected before requests are sent.
Load tests should normally target a controlled environment. NGINX caches each
full API URI for 30 seconds, and the API rate limiter allows 20 requests per
minute per client IP on each API instance. These settings can make a test
measure cache hits or rate limiting rather than database-read performance.
## Tech Stack ## Tech Stack
| Component | Technology | | Layer | Technology |
| ---------------- | ----------------------- | | :--- | :--- |
| Language | Go 1.23+ | | **Language** | Go 1.24 |
| Framework | Fiber v2 | | **HTTP Framework** | Fiber v2 (`valyala/fasthttp`) |
| ORM | GORM | | **Database** | PostgreSQL 17 (Primary + Read Replicas) |
| Database | PostgreSQL 15 | | **ORM & Routing** | GORM + `dbresolver` (R/W splitting) |
| Scraping | goquery | | **Reverse Proxy / Cache** | NGINX Stable (API caching & round-robin) |
| Load Balancer | NGINX | | **Edge Routing & SSL** | Traefik v3 (Let's Encrypt automated TLS) |
| Monitoring | Prometheus + Grafana | | **Monitoring** | Prometheus + Grafana |
| Docs | Swagger (swaggo) | | **Scraping** | `goquery` (HTML parsing) |
| Containerization | Docker + Docker Compose | | **Documentation** | Swagger 2.0 (`swaggo/swag`) |
| **Deployment** | Docker, Docker Compose, Coolify |
## Workflows