Initial commitmain

1 files changed, 287 insertions, 0 deletions

diff --git a/CLAUDE.md b/CLAUDE.md
new file mode 100644
index 0000000..7c16e88
--- /dev/null
+++ b/CLAUDE.md
@@ -0,0 +1,287 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+This is a Mihon & Aniyomi extensions repository aggregator that:
+
+- Fetches and mirrors extension repositories from multiple sources
+- Builds a static website to browse and add extensions
+- Uses R2 cache for fast extension data restoration
+- Mirrors repository to GitLab for backup
+- Automatically updates extensions every 4 hours via GitHub Actions
+
+## Development Commands
+
+```bash
+# Install dependencies
+bun install
+
+# Start development server with hot reload
+bun run dev
+
+# Update extensions from upstream sources
+bun run update
+
+# Build the static site
+bun run build
+
+# Preview the production build
+bun run preview
+
+# Format code with Prettier
+bun run format
+
+# Check code formatting
+bun run lint
+```
+
+## Architecture
+
+### Core Build System
+
+The build process uses Vite with SvelteKit and custom scripts:
+
+1. **`scripts/update.ts`**: Main extension update and data generation script
+    - Fetches extensions from upstream git repositories
+    - Reads `extensions.json` configuration at project root
+    - Checks remote git commit hashes for updates
+    - Supports multiple modes:
+        - `--generate-only`: Regenerate `data.json` without fetching updates or cache operations
+        - `--quick`: Fast mode that only updates `extensions.json` with new hashes (used in CI)
+        - `--no-cache`: Disable cache operations
+        - Default mode: Full update with cache operations
+    - Cache behavior:
+        - Restores `static/` directory from S3 cache (only in full mode)
+        - Uses manifest-based cache resolution with fallback prefixes
+        - Clones repositories to `tmp/` and copies configured files to `static/`
+        - Updates `extensions.json` with new commit hashes only after successful clone/copy
+        - Uploads updated `static/` directory to S3 cache
+        - Cleans up old cache entries (keeps most recent 10, max age 30 days)
+    - Generates `static/data.json` with extension metadata
+    - Sets `updated` output for CI/CD workflows (only when changes occur)
+
+2. **Build order**: `update --generate-only → vite build`
+    - First, `data.json` is regenerated from current extension state
+    - Then Vite builds the SvelteKit app into `dist/`
+    - Static assets from `static/` are included in the build
+
+### Extension Configuration
+
+Extensions are defined in `extensions.json` at the root with a nested structure by category:
+
+```json
+{
+  "mihon": {
+    "keiyoushi": {
+      "name": "Keiyoushi",
+      "source": "https://github.com/keiyoushi/extensions",
+      "path": "/keiyoushi/index.min.json",
+      "commit": "..."
+    },
+    "yuzono/manga": {
+      "name": "Yuzono Manga",
+      "source": "https://github.com/yuzono/manga-repo",
+      "path": "/yuzono/manga/index.min.json",
+      "commit": "..."
+    }
+  },
+  "aniyomi": {
+    "kohi-den": { ... }
+  }
+}
+```
+
+Each extension specifies:
+
+- `source`: Git repository URL
+- `path`: URL path for the extension index (can include subdirectories)
+- `commit`: Current tracked commit hash
+- The key (e.g., "keiyoushi", "yuzono/manga") is used as the directory name in `static/` and as part of the URL path
+
+### Files Copied from Extensions
+
+Defined in `scripts/config.ts` as `filesToCopy`:
+
+- `index.json` - Full extension index
+- `index.min.json` - Minified extension index
+- `repo.json` - Repository metadata
+- `apk/` - APK files directory
+- `icon/` - Icon files directory
+
+### Frontend
+
+The frontend is a SvelteKit static site built with Vite:
+
+- **SvelteKit**: Framework with static adapter (`@sveltejs/adapter-static`)
+- **Svelte 5**: Modern reactive UI framework
+- **TypeScript**: Type-safe component development
+- **Meilisearch**: Advanced search with filtering and faceting
+- **Prerendering**: All pages are prerendered at build time (`prerender = true`)
+
+**Frontend Structure**:
+
+- `src/routes/+layout.ts` - Loads `data.json` and provides it to all pages
+- `src/routes/+page.svelte` - Home page with extension categories
+- `src/routes/search/+page.svelte` - Search page with all extensions
+- `src/lib/components/` - Reusable UI components:
+    - `ExtensionCategory.svelte` - Category display with repos
+    - `ExtensionCard.svelte` - Individual repo card
+    - `ExtensionRow.svelte` - Table row for search results with NSFW badge
+    - `MirrorSelector.svelte` - Domain selection dropdown
+    - `Footer.svelte` - Page footer
+- `src/lib/stores/` - Svelte stores for state management
+- `src/lib/types.ts` - TypeScript type definitions
+- `src/lib/search/` - Search functionality with Meilisearch integration:
+    - `meilisearch.ts` - Meilisearch client and search operations
+    - `debounce.ts` - Search debouncing utilities
+    - `types.ts` - Search-specific type definitions
+    - `utils.ts` - Search helper functions
+- `src/app.css` - Global styles including NSFW badge styling
+- `src/app.html` - HTML template
+
+The frontend:
+
+1. Fetches `data.json` on initial load via `+layout.ts`
+2. Displays extension repositories grouped by category (mihon/aniyomi)
+3. Provides mirror domain selection for extension URLs
+4. Offers "Add Repo" links using `tachiyomi://` or `aniyomi://` protocols
+5. Search page allows browsing all individual extensions from all repos with advanced filtering
+6. Shows NSFW badge for extensions with adult content (when `nsfw: 1`)
+
+### Extension Data Structure
+
+Extensions fetched from upstream repositories contain:
+
+- `name`: Display name (e.g., "Tachiyomi: MangaDex")
+- `pkg`: Package identifier (e.g., "eu.kanade.tachiyomi.extension.en.mangadex")
+- `version`: Version string
+- `lang`: Language code (e.g., "en", "all")
+- `apk`: APK filename
+- `nsfw`: Integer flag (1 = NSFW content, 0 = safe)
+
+### Caching System
+
+The project uses S3-compatible storage (Cloudflare R2, Backblaze B2, AWS S3, etc.) for distributed caching of extension data with a sophisticated manifest-based system:
+
+- **Cache Storage**: Compressed tar.zst files stored in S3 bucket
+- **Compression**: Uses tar with zstd compression for fast compression/decompression
+- **Manifest System**: JSON manifest tracks all cache entries with metadata for resolution
+- **Distributed Locking**: Uses S3 metadata and conditional writes with instance IDs to coordinate updates
+- **Cache Validation**: Validates cache integrity using file checksums before restoration
+- **Automatic Cleanup**: Keeps most recent 10 caches, removes entries older than 30 days
+- **Cache Scripts**: Located in `scripts/`
+    - `cache.ts`: Main cache orchestration (restore and save operations with locking)
+    - `cache/files.ts`: Tar archive creation/extraction with zstd compression and checksum validation
+    - `cache/lock.ts`: Distributed lock implementation using S3 metadata and instance IDs
+    - `cache/manifest.ts`: Cache manifest management for tracking and resolving cache entries
+    - `cache/metadata.ts`: Cache metadata storage (checksums, timestamps) for validation
+    - `cache/s3.ts`: S3 client wrapper, cache resolution, and cleanup logic
+    - `cache/utils.ts`: Shared utilities and constants
+    - `cache/logger.ts`: Logging utilities for transfers and progress
+
+**S3 Configuration**:
+
+Required environment variables in `.env`:
+
+- `S3_ENDPOINT`: S3 endpoint URL
+    - Cloudflare R2: `https://<ACCOUNT_ID>.r2.cloudflarestorage.com`
+    - Backblaze B2: `https://s3.<REGION>.backblazeb2.com`
+    - AWS S3: `https://s3.<REGION>.amazonaws.com`
+- `S3_ACCESS_KEY_ID`: Access key ID
+- `S3_SECRET_ACCESS_KEY`: Secret access key
+- `S3_BUCKET_NAME`: Bucket name
+- `S3_REGION`: Region (optional, use "auto" for R2)
+
+**Cache Flow**:
+
+**Restore**:
+
+1. Resolve cache key using manifest (exact match or prefix fallback)
+2. Validate local cache using checksums (skip download if valid)
+3. Download tar.zst file from S3
+4. Extract to `static/` directory
+5. Update access timestamps in manifest
+
+**Save**:
+
+1. Acquire distributed lock using instance ID
+2. Compress `static/` directory to tar.zst with checksums
+3. Upload to S3 with streaming multipart upload
+4. Save metadata (checksums, file list) to S3
+5. Update manifest with new cache entry
+6. Clean up old cache entries (keep 10 most recent, max age 30 days)
+7. Release lock
+
+### Deployment
+
+The workflow is configured in `.github/workflows/update.yml`:
+
+- **GitLab Mirror**: Syncs entire repository to GitLab using SSH for backup and redundancy
+
+The workflow runs on:
+
+- Schedule: Every 4 hours (`0 */4 * * *`)
+- Manual: `workflow_dispatch`
+- Push to main branch (excluding bot commits)
+
+The workflow has two jobs:
+
+- `update`: Updates extensions using quick mode, commits changes to `extensions.json`
+- `sync-to-gitlab`: Mirrors repository to GitLab (runs after update when changes occur)
+
+Deployments and mirroring run when:
+
+- Extensions have updates (`updated=true` from `update.ts`)
+- Workflow is manually triggered (`workflow_dispatch`)
+
+## Important Patterns
+
+### Update Logic
+
+The update script (`scripts/update.ts`) uses a dual-config sync system:
+
+- `extensions.json` (root): The desired/target state
+- `static/data.json`: Contains the successfully synced state (what's actually downloaded)
+
+Update flow:
+
+1. Compare remote hash with synced hash (from `data.json`)
+2. If different, or if files are missing in `static/`, queue for update
+3. Clone and copy files for each extension
+4. **Only after successful clone/copy**, update `extensions.json` with new commit hash
+5. Generate new `data.json` with updated commit info
+6. Failed clones don't update hashes, ensuring retry on next run
+
+CI behavior:
+
+- In CI: only downloads if there are actual hash changes
+- Locally (non-CI): always downloads to restore missing files
+- Manual workflow triggers: force downloads regardless of hash changes
+- Sets `updated` output for CI/CD workflows based on successful updates only
+
+### CI Skip Pattern
+
+Commits from the workflow use `[skip ci]` to prevent recursive builds:
+
+```bash
+git commit -m "chore: update extensions.json"
+```
+
+### Code Standards
+
+- **Line Endings**: LF (Line Feed) is enforced project-wide via:
+    - `.gitattributes`: `* text=auto eol=lf`
+    - `.prettierrc`: `"endOfLine": "lf"`
+    - This ensures cross-platform consistency (Windows/Linux/CI)
+
+- **Formatting**: Prettier with:
+    - 4 spaces for indentation
+    - Single quotes
+    - No trailing commas
+    - 100 character line width
+
+### Configuration
+
+All deployable domains are listed in `scripts/config.ts` under `config.domains`. The frontend allows users to select which mirror to use for extension URLs.