Featured image of post Automated Financial Daily Reports with DuckDB: A Complete Implementation Guide

Automated Financial Daily Reports with DuckDB: A Complete Implementation Guide

Step-by-step guide to building a fully automated financial daily report system using DuckDB + Python. Learn how data analysts can generate 3000+ CNY/month in passive income through information processing.

In the data space, information processing is one of the most undervalued skills. Most data analysts confine their reporting abilities to internal presentations, never realizing this skill can be directly converted into subscription-based revenue. Today we’ll break down a real product: building a fully automated financial daily report with DuckDB, launching in 3 hours, generating a stable extra 3000+ CNY per month.

Architecture

1. Product Logic: What Are You Actually Selling?

Start with the pricing model before designing the technical architecture. A typical financial daily report product:

  • Target customers: Small investment firms, independent financial advisors, trading community members
  • Deliverable: Morning brief pushed to Telegram / WeChat groups
  • Content structure: Market overview → Hot sectors → Capital flow → Risk signals
  • Pricing reference: Monthly subscription 99-299 CNY, annual 999 CNY

The core value isn’t “data” — public data is everywhere. You sell processed insights. Raw data is free; insight services are paid.

2. Why DuckDB? Technical Architecture Comparison

Traditional approaches use PostgreSQL + Python scripts, but the operational cost is too high for individual developers. DuckDB’s core advantages:

DimensionTraditional (PostgreSQL)DuckDB Approach
Deployment2-4 hours (server + DB)30 minutes (code only)
Monthly cost50-200 CNY (cloud server)0 CNY
Analysis performanceRow storage, slower aggregationColumnar, million-row aggregation in seconds
Data ingestionRequires ETL pipeline firstNative HTTP / JSON / CSV support
EmbeddingClient connection requiredIn-process library, seamless Python/R/Node.js integration
Monthly costCloud server 50-200 CNYCompletely free

The entire system needs only one Python script + one GitHub Actions scheduled job — monthly cost = 0.

3. Project Structure

financial-daily-report/
├── report.py # Main program (fetch → analyze → push)
├── config.yaml # Configuration (API keys, Telegram Bot Token)
├── data/ # DuckDB database file
│ └── market.db
├── .github/workflows/
│ └── daily-cron.yml # GitHub Actions scheduled job
└── requirements.txt

4. Core Code Implementation

Step 1: Data Fetching and Storage

DuckDB can directly query JSON from HTTP endpoints, or read from CSV/Parquet files — no need to pre-load into a database:

import duckdb
import httpx
import yaml
from pathlib import Path
from datetime import datetime, timedelta
import json

CONFIG_PATH = Path(__file__).parent / "config.yaml"
with open(CONFIG_PATH) as f:
 cfg = yaml.safe_load(f)

DB_PATH = Path(__file__).parent / "data" / "market.db"
DB_PATH.parent.mkdir(parents=True, exist_ok=True)

def fetch_market_data() -> dict:
 """Fetch A-share market core indicators"""
 # Replace with real APIs like AKShare, Tushare in production
 return {
 "indices": [
 {"code": "SH000001", "name": "Shanghai Composite", "close": 3285.67, "pct": 0.42},
 {"code": "SZ399001", "name": "Shenzhen Component", "close": 10567.23, "pct": 0.18},
 {"code": "SZ399006", "name": "ChiNext Index", "close": 2156.89, "pct": -0.35},
 ],
 "sectors": [
 {"name": "Semiconductor", "pct": 2.14, "volume_ratio": 1.32},
 {"name": "New Energy", "pct": 1.87, "volume_ratio": 1.15},
 {"name": "Liquor", "pct": -0.52, "volume_ratio": 0.88},
 {"name": "AI Applications", "pct": 3.21, "volume_ratio": 1.67},
 ],
 "capital_flow": {
 "northbound": 12.5,
 "margin": -3.2,
 "main_force": 45.8,
 },
 }

def save_to_duckdb(data: dict) -> None:
 conn = duckdb.connect(str(DB_PATH))
 today = datetime.now().strftime("%Y-%m-%d")

 # Create tables if not exist, delete-then-insert for idempotent daily updates
 conn.execute("""
 CREATE TABLE IF NOT EXISTS daily_indices (
 date DATE,
 code VARCHAR,
 name VARCHAR,
 close DOUBLE,
 pct_change DOUBLE,
 created_at TIMESTAMP
 )
 """)
 conn.execute("DELETE FROM daily_indices WHERE date = ?", [today])
 for idx in data["indices"]:
 conn.execute(
 "INSERT INTO daily_indices VALUES (?, ?, ?, ?, ?, ?)",
 [today, idx["code"], idx["name"], idx["close"], idx["pct"], datetime.now()],
 )

 conn.execute("""
 CREATE TABLE IF NOT EXISTS daily_sectors (
 date DATE,
 sector_name VARCHAR,
 pct_change DOUBLE,
 volume_ratio DOUBLE,
 created_at TIMESTAMP
 )
 """)
 conn.execute("DELETE FROM daily_sectors WHERE date = ?", [today])
 for sec in data["sectors"]:
 conn.execute(
 "INSERT INTO daily_sectors VALUES (?, ?, ?, ?, ?)",
 [today, sec["name"], sec["pct"], sec["volume_ratio"], datetime.now()],
 )

 conn.execute("""
 CREATE TABLE IF NOT EXISTS daily_capital (
 date DATE,
 northbound DOUBLE,
 margin_change DOUBLE,
 main_force DOUBLE,
 created_at TIMESTAMP
 )
 """)
 conn.execute("DELETE FROM daily_capital WHERE date = ?", [today])
 conn.execute(
 "INSERT INTO daily_capital VALUES (?, ?, ?, ?, ?)",
 [today, data["capital_flow"]["northbound"],
 data["capital_flow"]["margin"], data["capital_flow"]["main_force"],
 datetime.now()],
 )

 conn.commit()
 conn.close()

Step 2: SQL-Driven Intelligence Engine

This is the core competitive advantage — using SQL to perform trend detection, sector ranking, and capital flow signal analysis:

def analyze_history(days: int = 5) -> dict:
 """Generate trend insights from historical data"""
 conn = duckdb.connect(str(DB_PATH))

 # Shanghai Composite trend over N days
 sh_trend = conn.execute("""
 SELECT date, close, pct_change
 FROM daily_indices
 WHERE code = 'SH000001'
 AND date >= CURRENT_DATE - INTERVAL ? DAY
 ORDER BY date DESC
 """, [days]).fetchall()

 # Sector heat ranking (filter by volume ratio > 1.0)
 hot_sectors = conn.execute("""
 SELECT sector_name, pct_change, volume_ratio
 FROM daily_sectors
 WHERE date = CURRENT_DATE
 AND volume_ratio > 1.0
 ORDER BY pct_change DESC
 LIMIT 3
 """).fetchall()

 # N-day capital flow summary
 capital_summary = conn.execute("""
 SELECT
 SUM(northbound) AS total_north,
 AVG(main_force) AS avg_main,
 CASE
 WHEN SUM(northbound) > 0 THEN '🟢 Foreign Capital Inflow'
 ELSE '🔴 Foreign Capital Outflow'
 END AS signal
 FROM daily_capital
 WHERE date >= CURRENT_DATE - INTERVAL ? DAY
 """, [days]).fetchone()

 conn.close()
 return {
 "sh_trend": sh_trend,
 "hot_sectors": hot_sectors,
 "capital": capital_summary,
 }

Key points in this SQL engine:

  1. CURRENT_DATE - INTERVAL ? DAY: DuckDB supports parameterized date arithmetic, avoiding string concatenation
  2. CASE WHEN signal generation: Conditional logic directly in SQL is clearer than post-processing in Python
  3. Delete-then-insert pattern: DELETE WHERE date = ? ensures idempotent daily updates

Step 3: Report Text Generation

def build_report(data: dict, insight: dict) -> str:
 today = datetime.now().strftime("%b %d")
 trend = insight["sh_trend"]
 last_close = trend[0][1] if trend else "?"
 prev_pct = trend[1][2] if len(trend) > 1 else 0

 up_days = sum(1 for row in trend if row[2] > 0)
 trend_text = f"Up {up_days}/{len(trend)} days" if trend else "No historical data"

 sector_lines = []
 for name, pct, vr in insight["hot_sectors"]:
 sector_lines.append(f" · {name} +{pct:.2f}% (VolRatio {vr:.2f})")
 sector_text = "\n".join(sector_lines) or " No high-volume sectors"

 capital = insight["capital"]
 capital_text = f"{capital[2]}, N-day northbound累计{'+' if capital[0]>0 else ''}{capital[0]:.1f}B"

 report = f"""📊 【{today} Financial Daily】

━━━ I. Market Overview ━━━
Shanghai Comp {last_close:.2f} ({'↑' if prev_pct>=0 else '↓'}{abs(prev_pct):.2f}%)
{trend_text}

━━━ II. Hot Sectors TOP3 ━━━
{sector_text}

━━━ III. Capital Flow ━━━
{capital_text}

━━━ IV. Tomorrow's Signal ━━━
{generate_signal(insight)}

📌 Data source: Public market data, for reference only"""
 return report

5. Automated Deployment: GitHub Actions + Cron

Use GitHub Actions’ cron feature for hands-free pushing — completely free:

# .github/workflows/daily-cron.yml
name: Daily Financial Report

on:
 schedule:
 - cron: '0 22 * * *' # 06:00 Beijing time (UTC 18:00)
 workflow_dispatch: # Also supports manual trigger

jobs:
 run-report:
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v4

 - name: Set up Python
 uses: actions/setup-python@v5
 with:
 python-version: '3.11'

 - name: Install dependencies
 run: pip install -r requirements.txt

 - name: Run report generator
 env:
 TG_BOT_TOKEN: ${{ secrets.TG_BOT_TOKEN }}
 TG_CHAT_ID: ${{ secrets.TG_CHAT_ID }}
 run: python report.py

 - name: Commit and push results
 run: |
 git config user.name "CI Bot"
 git config user.email "ci@duckdblab.org"
 git add data/
 git commit -m "Auto update: $(date +%Y-%m-%d)" || echo "No changes"
 git push

Store sensitive configs in config.yaml:

# config.yaml
telegram:
 bot_token: "YOUR_BOT_TOKEN"
 chat_id: "YOUR_CHAT_ID"

data_source: "akshare" # or "tushare", "mock"

6. Efficiency Comparison: Traditional vs DuckDB

MetricPostgreSQL ApproachDuckDB Approach
Setup time2-4 hours (server + DB)30 minutes (code only)
Monthly cost50-200 CNY (cloud)0 CNY
Data storage (1 month)~1.5M rows (3 tables × 5 years)Single file < 5MB
Query latencyRequires network connectionLocal millisecond-level
Data recoveryRequires backup strategyCopy .db file directly
Scaling difficultyVertical scaling neededJust append historical data

8. Expansion: From Daily Report to Data Product

Once you’ve validated the daily report loop, expand further:

  • Multi-market coverage: A-shares → Hong Kong → US markets, one codebase, three revenue streams
  • Real-time alerts: Add price breakout and capital anomaly detection for instant push notifications
  • Historical backtesting: Use DuckDB’s Time Travel to backtest strategies
  • Visual dashboards: Integrate with Evidence or Streamlit for auto-generated HTML reports

📖 详细图文教程见 duckdblab.org

💡 更多 DuckDB 实战技巧 → duckdblab.org

📺 Watch video tutorials → Olap Studio YouTube

Subscribe for more DuckDB & AI automation tutorials