diff options
Diffstat (limited to '')
| -rw-r--r-- | CLAUDE.md | 287 |
1 files changed, 287 insertions, 0 deletions
diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..7c16e88 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,287 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project Overview + +This is a Mihon & Aniyomi extensions repository aggregator that: + +- Fetches and mirrors extension repositories from multiple sources +- Builds a static website to browse and add extensions +- Uses R2 cache for fast extension data restoration +- Mirrors repository to GitLab for backup +- Automatically updates extensions every 4 hours via GitHub Actions + +## Development Commands + +```bash +# Install dependencies +bun install + +# Start development server with hot reload +bun run dev + +# Update extensions from upstream sources +bun run update + +# Build the static site +bun run build + +# Preview the production build +bun run preview + +# Format code with Prettier +bun run format + +# Check code formatting +bun run lint +``` + +## Architecture + +### Core Build System + +The build process uses Vite with SvelteKit and custom scripts: + +1. **`scripts/update.ts`**: Main extension update and data generation script + - Fetches extensions from upstream git repositories + - Reads `extensions.json` configuration at project root + - Checks remote git commit hashes for updates + - Supports multiple modes: + - `--generate-only`: Regenerate `data.json` without fetching updates or cache operations + - `--quick`: Fast mode that only updates `extensions.json` with new hashes (used in CI) + - `--no-cache`: Disable cache operations + - Default mode: Full update with cache operations + - Cache behavior: + - Restores `static/` directory from S3 cache (only in full mode) + - Uses manifest-based cache resolution with fallback prefixes + - Clones repositories to `tmp/` and copies configured files to `static/` + - Updates `extensions.json` with new commit hashes only after successful clone/copy + - Uploads updated `static/` directory to S3 cache + - Cleans up old cache entries (keeps most recent 10, max age 30 days) + - Generates `static/data.json` with extension metadata + - Sets `updated` output for CI/CD workflows (only when changes occur) + +2. **Build order**: `update --generate-only → vite build` + - First, `data.json` is regenerated from current extension state + - Then Vite builds the SvelteKit app into `dist/` + - Static assets from `static/` are included in the build + +### Extension Configuration + +Extensions are defined in `extensions.json` at the root with a nested structure by category: + +```json +{ + "mihon": { + "keiyoushi": { + "name": "Keiyoushi", + "source": "https://github.com/keiyoushi/extensions", + "path": "/keiyoushi/index.min.json", + "commit": "..." + }, + "yuzono/manga": { + "name": "Yuzono Manga", + "source": "https://github.com/yuzono/manga-repo", + "path": "/yuzono/manga/index.min.json", + "commit": "..." + } + }, + "aniyomi": { + "kohi-den": { ... } + } +} +``` + +Each extension specifies: + +- `source`: Git repository URL +- `path`: URL path for the extension index (can include subdirectories) +- `commit`: Current tracked commit hash +- The key (e.g., "keiyoushi", "yuzono/manga") is used as the directory name in `static/` and as part of the URL path + +### Files Copied from Extensions + +Defined in `scripts/config.ts` as `filesToCopy`: + +- `index.json` - Full extension index +- `index.min.json` - Minified extension index +- `repo.json` - Repository metadata +- `apk/` - APK files directory +- `icon/` - Icon files directory + +### Frontend + +The frontend is a SvelteKit static site built with Vite: + +- **SvelteKit**: Framework with static adapter (`@sveltejs/adapter-static`) +- **Svelte 5**: Modern reactive UI framework +- **TypeScript**: Type-safe component development +- **Meilisearch**: Advanced search with filtering and faceting +- **Prerendering**: All pages are prerendered at build time (`prerender = true`) + +**Frontend Structure**: + +- `src/routes/+layout.ts` - Loads `data.json` and provides it to all pages +- `src/routes/+page.svelte` - Home page with extension categories +- `src/routes/search/+page.svelte` - Search page with all extensions +- `src/lib/components/` - Reusable UI components: + - `ExtensionCategory.svelte` - Category display with repos + - `ExtensionCard.svelte` - Individual repo card + - `ExtensionRow.svelte` - Table row for search results with NSFW badge + - `MirrorSelector.svelte` - Domain selection dropdown + - `Footer.svelte` - Page footer +- `src/lib/stores/` - Svelte stores for state management +- `src/lib/types.ts` - TypeScript type definitions +- `src/lib/search/` - Search functionality with Meilisearch integration: + - `meilisearch.ts` - Meilisearch client and search operations + - `debounce.ts` - Search debouncing utilities + - `types.ts` - Search-specific type definitions + - `utils.ts` - Search helper functions +- `src/app.css` - Global styles including NSFW badge styling +- `src/app.html` - HTML template + +The frontend: + +1. Fetches `data.json` on initial load via `+layout.ts` +2. Displays extension repositories grouped by category (mihon/aniyomi) +3. Provides mirror domain selection for extension URLs +4. Offers "Add Repo" links using `tachiyomi://` or `aniyomi://` protocols +5. Search page allows browsing all individual extensions from all repos with advanced filtering +6. Shows NSFW badge for extensions with adult content (when `nsfw: 1`) + +### Extension Data Structure + +Extensions fetched from upstream repositories contain: + +- `name`: Display name (e.g., "Tachiyomi: MangaDex") +- `pkg`: Package identifier (e.g., "eu.kanade.tachiyomi.extension.en.mangadex") +- `version`: Version string +- `lang`: Language code (e.g., "en", "all") +- `apk`: APK filename +- `nsfw`: Integer flag (1 = NSFW content, 0 = safe) + +### Caching System + +The project uses S3-compatible storage (Cloudflare R2, Backblaze B2, AWS S3, etc.) for distributed caching of extension data with a sophisticated manifest-based system: + +- **Cache Storage**: Compressed tar.zst files stored in S3 bucket +- **Compression**: Uses tar with zstd compression for fast compression/decompression +- **Manifest System**: JSON manifest tracks all cache entries with metadata for resolution +- **Distributed Locking**: Uses S3 metadata and conditional writes with instance IDs to coordinate updates +- **Cache Validation**: Validates cache integrity using file checksums before restoration +- **Automatic Cleanup**: Keeps most recent 10 caches, removes entries older than 30 days +- **Cache Scripts**: Located in `scripts/` + - `cache.ts`: Main cache orchestration (restore and save operations with locking) + - `cache/files.ts`: Tar archive creation/extraction with zstd compression and checksum validation + - `cache/lock.ts`: Distributed lock implementation using S3 metadata and instance IDs + - `cache/manifest.ts`: Cache manifest management for tracking and resolving cache entries + - `cache/metadata.ts`: Cache metadata storage (checksums, timestamps) for validation + - `cache/s3.ts`: S3 client wrapper, cache resolution, and cleanup logic + - `cache/utils.ts`: Shared utilities and constants + - `cache/logger.ts`: Logging utilities for transfers and progress + +**S3 Configuration**: + +Required environment variables in `.env`: + +- `S3_ENDPOINT`: S3 endpoint URL + - Cloudflare R2: `https://<ACCOUNT_ID>.r2.cloudflarestorage.com` + - Backblaze B2: `https://s3.<REGION>.backblazeb2.com` + - AWS S3: `https://s3.<REGION>.amazonaws.com` +- `S3_ACCESS_KEY_ID`: Access key ID +- `S3_SECRET_ACCESS_KEY`: Secret access key +- `S3_BUCKET_NAME`: Bucket name +- `S3_REGION`: Region (optional, use "auto" for R2) + +**Cache Flow**: + +**Restore**: + +1. Resolve cache key using manifest (exact match or prefix fallback) +2. Validate local cache using checksums (skip download if valid) +3. Download tar.zst file from S3 +4. Extract to `static/` directory +5. Update access timestamps in manifest + +**Save**: + +1. Acquire distributed lock using instance ID +2. Compress `static/` directory to tar.zst with checksums +3. Upload to S3 with streaming multipart upload +4. Save metadata (checksums, file list) to S3 +5. Update manifest with new cache entry +6. Clean up old cache entries (keep 10 most recent, max age 30 days) +7. Release lock + +### Deployment + +The workflow is configured in `.github/workflows/update.yml`: + +- **GitLab Mirror**: Syncs entire repository to GitLab using SSH for backup and redundancy + +The workflow runs on: + +- Schedule: Every 4 hours (`0 */4 * * *`) +- Manual: `workflow_dispatch` +- Push to main branch (excluding bot commits) + +The workflow has two jobs: + +- `update`: Updates extensions using quick mode, commits changes to `extensions.json` +- `sync-to-gitlab`: Mirrors repository to GitLab (runs after update when changes occur) + +Deployments and mirroring run when: + +- Extensions have updates (`updated=true` from `update.ts`) +- Workflow is manually triggered (`workflow_dispatch`) + +## Important Patterns + +### Update Logic + +The update script (`scripts/update.ts`) uses a dual-config sync system: + +- `extensions.json` (root): The desired/target state +- `static/data.json`: Contains the successfully synced state (what's actually downloaded) + +Update flow: + +1. Compare remote hash with synced hash (from `data.json`) +2. If different, or if files are missing in `static/`, queue for update +3. Clone and copy files for each extension +4. **Only after successful clone/copy**, update `extensions.json` with new commit hash +5. Generate new `data.json` with updated commit info +6. Failed clones don't update hashes, ensuring retry on next run + +CI behavior: + +- In CI: only downloads if there are actual hash changes +- Locally (non-CI): always downloads to restore missing files +- Manual workflow triggers: force downloads regardless of hash changes +- Sets `updated` output for CI/CD workflows based on successful updates only + +### CI Skip Pattern + +Commits from the workflow use `[skip ci]` to prevent recursive builds: + +```bash +git commit -m "chore: update extensions.json" +``` + +### Code Standards + +- **Line Endings**: LF (Line Feed) is enforced project-wide via: + - `.gitattributes`: `* text=auto eol=lf` + - `.prettierrc`: `"endOfLine": "lf"` + - This ensures cross-platform consistency (Windows/Linux/CI) + +- **Formatting**: Prettier with: + - 4 spaces for indentation + - Single quotes + - No trailing commas + - 100 character line width + +### Configuration + +All deployable domains are listed in `scripts/config.ts` under `config.domains`. The frontend allows users to select which mirror to use for extension URLs. |