feat: smart git URL parsing for browser URLs
- Add git URL parsing utilities (extract_base_repo_url, parse_git_url) - Support GitHub, GitLab, Bitbucket browser URL detection - Add /projects/repositories/parse-url endpoint - Enhance repository creation to detect browser URLs and suggest corrections - Add real-time URL validation in frontend with debouncing - Show visual indicators (green/yellow/red) for URL validity - Display inline suggestions with 'Use Suggested' button - Add comprehensive unit tests for URL parsing - Quality gates: ruff ✓, mypy ✓, typecheck ✓, lint ✓, build ✓
This commit is contained in:
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-05-19
|
||||
@@ -0,0 +1,154 @@
|
||||
# Smart Git URL Parsing - Design
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
User pastes URL → Frontend validates → Backend validates → Clone repo
|
||||
↓ ↓
|
||||
Show suggestions Parse & suggest
|
||||
```
|
||||
|
||||
## URL Detection Logic
|
||||
|
||||
### Patterns to Detect
|
||||
|
||||
1. **GitHub/GitLab/Bitbucket browser URLs**
|
||||
- `https://github.com/owner/repo/tree/branch-name`
|
||||
- `https://github.com/owner/repo/blob/branch/path/to/file`
|
||||
- `https://github.com/owner/repo/pull/123`
|
||||
- `https://gitlab.com/owner/repo/-/tree/branch`
|
||||
- `https://bitbucket.org/owner/repo/src/branch/`
|
||||
|
||||
2. **URLs with query parameters**
|
||||
- `https://github.com/owner/repo?tab=readme-ov-file`
|
||||
- `https://github.com/owner/repo.git?branch=develop`
|
||||
|
||||
3. **Valid clone URLs (should pass through)**
|
||||
- `https://github.com/owner/repo.git`
|
||||
- `git@github.com:owner/repo.git`
|
||||
- `https://github.com/owner/repo` (without .git)
|
||||
|
||||
### URL Parsing Rules
|
||||
|
||||
```python
|
||||
def extract_base_repo_url(url: str) -> str | None:
|
||||
"""Extract base repository URL from a browser/git URL.
|
||||
|
||||
Examples:
|
||||
https://github.com/user/repo/tree/main → https://github.com/user/repo.git
|
||||
https://github.com/user/repo.git → https://github.com/user/repo.git
|
||||
git@github.com:user/repo.git → git@github.com:user/repo.git
|
||||
https://gitlab.com/user/repo/-/blob/main/README.md → https://gitlab.com/user/repo.git
|
||||
|
||||
Returns None if URL doesn't match known patterns.
|
||||
"""
|
||||
...
|
||||
```
|
||||
|
||||
**Algorithm:**
|
||||
1. Remove query parameters
|
||||
2. Detect host (github.com, gitlab.com, bitbucket.org, etc.)
|
||||
3. For GitHub: Remove `/tree/*`, `/blob/*`, `/pull/*`, `/issues/*` paths
|
||||
4. For GitLab: Remove `/-/tree/*`, `/-/blob/*` paths
|
||||
5. For Bitbucket: Remove `/src/*` paths
|
||||
6. Ensure `.git` suffix
|
||||
7. Return cleaned URL or None
|
||||
|
||||
## API Changes
|
||||
|
||||
### POST /repositories (enhanced)
|
||||
|
||||
**Request Body:**
|
||||
```json
|
||||
{
|
||||
"project_id": "uuid",
|
||||
"name": "my-repo",
|
||||
"remote_url": "https://github.com/user/repo/tree/main",
|
||||
"is_mirror": false
|
||||
}
|
||||
```
|
||||
|
||||
**New Response for Non-Repo URLs (422):**
|
||||
```json
|
||||
{
|
||||
"detail": "URL appears to be a browser URL, not a git clone URL",
|
||||
"suggested_url": "https://github.com/user/repo.git",
|
||||
"original_url": "https://github.com/user/repo/tree/main",
|
||||
"needs_confirmation": true
|
||||
}
|
||||
```
|
||||
|
||||
### New Endpoint: POST /repositories/parse-url
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"url": "https://github.com/user/repo/tree/main"
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"original_url": "https://github.com/user/repo/tree/main",
|
||||
"base_url": "https://github.com/user/repo.git",
|
||||
"is_valid_repo_url": false,
|
||||
"needs_parsing": true,
|
||||
"message": "This looks like a browser URL. Did you mean to clone https://github.com/user/repo.git?"
|
||||
}
|
||||
```
|
||||
|
||||
## Frontend Flow
|
||||
|
||||
### Repository Creation Dialog (Enhanced)
|
||||
|
||||
1. **User pastes URL**
|
||||
2. **Frontend calls `/repositories/parse-url`** (debounced)
|
||||
3. **If URL needs parsing:**
|
||||
- Show yellow warning indicator
|
||||
- Display: "This looks like a browser URL"
|
||||
- Show suggested URL with "Use this instead" button
|
||||
- Allow user to proceed with original URL anyway
|
||||
4. **If URL is valid:**
|
||||
- Show green checkmark
|
||||
- Proceed normally
|
||||
5. **User clicks "Create"**
|
||||
6. **If backend returns 422 with suggestion:**
|
||||
- Show confirmation dialog with suggested URL
|
||||
- Options: "Use suggested URL", "Use original", "Cancel"
|
||||
|
||||
### UI Components
|
||||
|
||||
**URLInput Component:**
|
||||
- Input field with validation status icon
|
||||
- Shows inline suggestions when URL is detected as browser URL
|
||||
- Green/yellow/red border based on validation
|
||||
|
||||
**URLCorrectionDialog Component:**
|
||||
- Modal dialog for confirming URL correction
|
||||
- Shows before/after comparison
|
||||
- Clear action buttons
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. **Backend utilities** - URL parsing functions with tests
|
||||
2. **Backend endpoint** - `/repositories/parse-url`
|
||||
3. **Backend validation** - Enhanced POST /repositories with suggestion response
|
||||
4. **Frontend URL input** - Enhanced input with validation feedback
|
||||
5. **Frontend dialog** - Confirmation dialog for URL corrections
|
||||
6. **Integration** - Wire up parse-url endpoint to frontend
|
||||
7. **Tests** - Unit tests for URL parsing, integration tests for flow
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Invalid URLs
|
||||
- Completely malformed URLs: Return 400 with clear message
|
||||
- Unsupported hosts: Return 400 with "Unsupported git host"
|
||||
- Private repos (auth needed): Return 401/403 with auth instructions
|
||||
- Non-existent repos: Return 404 (from git clone failure)
|
||||
|
||||
### Clone Failures
|
||||
- Network issues: Retry with exponential backoff
|
||||
- Auth required: Prompt for credentials
|
||||
- Large repos: Show progress indicator
|
||||
- Timeout: Increase timeout for large repos
|
||||
@@ -0,0 +1,43 @@
|
||||
# Smart Git URL Parsing for Repository Creation
|
||||
|
||||
## Problem
|
||||
|
||||
When users add a new repository, they often paste a full browser URL that includes branch names, file paths, or query parameters instead of a clean repository URL. This causes the clone operation to fail with unclear error messages.
|
||||
|
||||
**Examples of problematic URLs:**
|
||||
- `https://github.com/user/repo/tree/main` (includes branch path)
|
||||
- `https://github.com/user/repo/blob/main/README.md` (includes file path)
|
||||
- `https://github.com/user/repo?tab=readme-ov-file` (includes query params)
|
||||
- `https://github.com/user/repo/pull/123` (includes PR path)
|
||||
|
||||
**Current behavior:** The backend attempts to clone the exact URL, which fails with "fatal: repository not found" or similar errors.
|
||||
|
||||
**User confusion:** Users don't understand why the clone failed since the URL works in their browser.
|
||||
|
||||
## Solution
|
||||
|
||||
Implement smart URL parsing that:
|
||||
|
||||
1. **Detects non-repo URLs**: Recognizes when a URL contains paths like `/tree/`, `/blob/`, `/pull/`, or query parameters
|
||||
2. **Extracts base repo URL**: Strips away branch names, file paths, query parameters to get `https://host/owner/repo.git`
|
||||
3. **Suggests correction**: Shows the user the extracted base URL and asks for confirmation
|
||||
4. **Improves clone handling**: Handles edge cases and provides clear error messages
|
||||
|
||||
## Benefits
|
||||
|
||||
- **Better UX**: Users get helpful suggestions instead of cryptic errors
|
||||
- **Fewer support issues**: Self-service correction reduces confusion
|
||||
- **More robust**: Handles common copy-paste mistakes automatically
|
||||
- **Educational**: Teaches users what a proper git URL looks like
|
||||
|
||||
## Scope
|
||||
|
||||
### Backend
|
||||
- URL parsing utilities to detect and extract base repo URLs
|
||||
- Enhanced validation in repository creation endpoint
|
||||
- Clear error messages for unsupported URLs
|
||||
|
||||
### Frontend
|
||||
- UI dialog to show URL correction suggestions
|
||||
- Option to accept or edit the suggested URL
|
||||
- Visual indicator for URL validation status
|
||||
@@ -0,0 +1,206 @@
|
||||
# Smart Git URL Parsing Specification
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional Requirements
|
||||
|
||||
1. **URL Detection**: Detect when a provided URL is a browser URL rather than a git clone URL
|
||||
2. **URL Extraction**: Extract the base repository URL from browser URLs
|
||||
3. **User Confirmation**: Show extracted URL to user and ask for confirmation
|
||||
4. **Flexible Input**: Allow users to proceed with original URL if they prefer
|
||||
5. **Multiple Hosts**: Support GitHub, GitLab, Bitbucket, and generic git hosts
|
||||
|
||||
### Non-Functional Requirements
|
||||
|
||||
1. **Performance**: URL parsing should be instant (< 100ms)
|
||||
2. **Accuracy**: Should correctly identify 95%+ of browser URLs
|
||||
3. **User Experience**: Clear, helpful messages without technical jargon
|
||||
4. **Backward Compatibility**: Existing valid git URLs should continue to work
|
||||
|
||||
## API Specification
|
||||
|
||||
### POST /git-repositories/parse-url
|
||||
Parse a URL and determine if it's a valid clone URL or needs correction.
|
||||
|
||||
**Request Body:**
|
||||
```json
|
||||
{
|
||||
"url": "https://github.com/user/repo/tree/main"
|
||||
}
|
||||
```
|
||||
|
||||
**Response 200:**
|
||||
```json
|
||||
{
|
||||
"original_url": "https://github.com/user/repo/tree/main",
|
||||
"base_url": "https://github.com/user/repo.git",
|
||||
"is_valid_clone_url": false,
|
||||
"needs_parsing": true,
|
||||
"host": "github.com",
|
||||
"message": "This URL contains a branch path. The repository URL is: https://github.com/user/repo.git"
|
||||
}
|
||||
```
|
||||
|
||||
**Response 200 (already valid):**
|
||||
```json
|
||||
{
|
||||
"original_url": "https://github.com/user/repo.git",
|
||||
"base_url": "https://github.com/user/repo.git",
|
||||
"is_valid_clone_url": true,
|
||||
"needs_parsing": false,
|
||||
"host": "github.com",
|
||||
"message": "Valid git repository URL"
|
||||
}
|
||||
```
|
||||
|
||||
### Enhanced POST /projects/{project_id}/repositories
|
||||
Enhanced to return suggestions when URL needs parsing.
|
||||
|
||||
**New 422 Response:**
|
||||
```json
|
||||
{
|
||||
"detail": "The provided URL appears to be a browser URL, not a git clone URL",
|
||||
"suggested_url": "https://github.com/user/repo.git",
|
||||
"original_url": "https://github.com/user/repo/tree/main",
|
||||
"error_code": "URL_NEEDS_PARSING"
|
||||
}
|
||||
```
|
||||
|
||||
**Request Body (with force flag):**
|
||||
```json
|
||||
{
|
||||
"name": "my-repo",
|
||||
"remote_url": "https://github.com/user/repo/tree/main",
|
||||
"is_mirror": false,
|
||||
"force_original_url": true
|
||||
}
|
||||
```
|
||||
|
||||
## Data Model
|
||||
|
||||
### URLParseResult
|
||||
```python
|
||||
class URLParseResult:
|
||||
original_url: str
|
||||
base_url: str | None
|
||||
is_valid_clone_url: bool
|
||||
needs_parsing: bool
|
||||
host: str | None
|
||||
message: str
|
||||
error_code: str | None
|
||||
```
|
||||
|
||||
## URL Parsing Rules
|
||||
|
||||
### Supported Hosts
|
||||
- github.com
|
||||
- gitlab.com
|
||||
- bitbucket.org
|
||||
- Any custom domain with git hosting
|
||||
|
||||
### GitHub URL Patterns
|
||||
```
|
||||
https://github.com/{owner}/{repo} → valid
|
||||
https://github.com/{owner}/{repo}.git → valid
|
||||
https://github.com/{owner}/{repo}/tree/{branch} → extract base
|
||||
https://github.com/{owner}/{repo}/blob/{branch}/{path} → extract base
|
||||
https://github.com/{owner}/{repo}/pull/{number} → extract base
|
||||
https://github.com/{owner}/{repo}/issues/{number} → extract base
|
||||
https://github.com/{owner}/{repo}/actions → extract base
|
||||
```
|
||||
|
||||
### GitLab URL Patterns
|
||||
```
|
||||
https://gitlab.com/{owner}/{repo} → valid
|
||||
https://gitlab.com/{owner}/{repo}.git → valid
|
||||
https://gitlab.com/{owner}/{repo}/-/tree/{branch} → extract base
|
||||
https://gitlab.com/{owner}/{repo}/-/blob/{branch}/{path} → extract base
|
||||
https://gitlab.com/{owner}/{repo}/-/merge_requests/{number} → extract base
|
||||
```
|
||||
|
||||
### Bitbucket URL Patterns
|
||||
```
|
||||
https://bitbucket.org/{owner}/{repo} → valid
|
||||
https://bitbucket.org/{owner}/{repo}.git → valid
|
||||
https://bitbucket.org/{owner}/{repo}/src/{branch} → extract base
|
||||
```
|
||||
|
||||
### Extraction Algorithm
|
||||
1. Parse URL components (scheme, netloc, path, query)
|
||||
2. Remove query parameters entirely
|
||||
3. Split path by `/`
|
||||
4. Remove trailing segments that indicate non-repo paths:
|
||||
- `tree/*`, `blob/*`, `pull/*`, `issues/*`, `actions`
|
||||
- `-/tree/*`, `-/blob/*`, `-/merge_requests/*`
|
||||
- `src/*`
|
||||
5. Reconstruct URL with remaining path
|
||||
6. Add `.git` suffix if missing (for HTTPS URLs)
|
||||
7. Return extracted URL
|
||||
|
||||
## Frontend Specification
|
||||
|
||||
### URL Input Component
|
||||
- Input field for repository URL
|
||||
- Real-time validation (debounced 300ms)
|
||||
- Visual indicators:
|
||||
- 🟢 Green border: Valid git URL
|
||||
- 🟡 Yellow border: Browser URL detected, suggestion shown
|
||||
- 🔴 Red border: Invalid/malformed URL
|
||||
- Inline suggestion banner below input:
|
||||
```
|
||||
⚠️ This looks like a browser URL
|
||||
Suggested: https://github.com/user/repo.git
|
||||
[Use Suggested] [Keep Original]
|
||||
```
|
||||
|
||||
### Confirmation Dialog
|
||||
Shown when backend returns 422 with suggestion:
|
||||
```
|
||||
┌─────────────────────────────────────┐
|
||||
│ URL Correction Suggestion │
|
||||
├─────────────────────────────────────┤
|
||||
│ │
|
||||
│ The URL you entered appears to be │
|
||||
│ a browser URL, not a git clone URL. │
|
||||
│ │
|
||||
│ Original: │
|
||||
│ https://github.com/user/repo/tree/main│
|
||||
│ │
|
||||
│ Suggested repository URL: │
|
||||
│ https://github.com/user/repo.git │
|
||||
│ │
|
||||
│ [Use Suggested URL] [Use Original] │
|
||||
│ [Cancel] │
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Error Codes
|
||||
|
||||
| Error Code | Description | User Message |
|
||||
|------------|-------------|--------------|
|
||||
| URL_NEEDS_PARSING | Browser URL detected | "This looks like a browser URL. Did you mean: {suggested_url}?" |
|
||||
| INVALID_URL | Malformed URL | "Please enter a valid URL" |
|
||||
| UNSUPPORTED_HOST | Unknown git host | "This git host is not supported" |
|
||||
| CLONE_FAILED | Git clone failed | "Failed to clone repository: {error}" |
|
||||
| AUTH_REQUIRED | Private repo, need auth | "This repository requires authentication" |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests (URL Parsing)
|
||||
- Test all GitHub URL patterns
|
||||
- Test all GitLab URL patterns
|
||||
- Test all Bitbucket URL patterns
|
||||
- Test valid URLs pass through unchanged
|
||||
- Test edge cases (subgroups, nested paths, etc.)
|
||||
|
||||
### Integration Tests
|
||||
- Test parse-url endpoint with various URLs
|
||||
- Test repository creation with browser URL (should suggest correction)
|
||||
- Test repository creation with force flag
|
||||
- Test clone operation with corrected URL
|
||||
|
||||
### Frontend Tests
|
||||
- Test URL input validation states
|
||||
- Test suggestion banner display
|
||||
- Test confirmation dialog flow
|
||||
- Test acceptance/rejection of suggestions
|
||||
@@ -0,0 +1,61 @@
|
||||
# Smart Git URL Parsing - Tasks
|
||||
|
||||
## Phase 1: Backend URL Parsing
|
||||
|
||||
- [ ] **Task 1.1**: Create URL parsing utilities
|
||||
- Create `src/utils/git_url_parser.py`
|
||||
- Implement `extract_base_repo_url()` function
|
||||
- Support GitHub, GitLab, Bitbucket patterns
|
||||
- Handle query parameters, branch paths, file paths
|
||||
- Add comprehensive unit tests
|
||||
|
||||
- [ ] **Task 1.2**: Create URL validation endpoint
|
||||
- Add `POST /git-repositories/parse-url` endpoint
|
||||
- Returns URLParseResult with original, base, validation status
|
||||
- Add tests for endpoint
|
||||
|
||||
- [ ] **Task 1.3**: Enhance repository creation endpoint
|
||||
- Update `POST /projects/{project_id}/repositories`
|
||||
- Detect browser URLs and return 422 with suggestion
|
||||
- Add `force_original_url` flag to bypass suggestion
|
||||
- Update response models
|
||||
|
||||
## Phase 2: Frontend Implementation
|
||||
|
||||
- [ ] **Task 2.1**: Create API client for URL parsing
|
||||
- Add `parseGitUrl()` function to `api/git_repositories.ts`
|
||||
- Add types for URLParseResult
|
||||
|
||||
- [ ] **Task 2.2**: Enhance repository creation form
|
||||
- Add real-time URL validation to input field
|
||||
- Show visual indicators (green/yellow/red)
|
||||
- Display inline suggestion banner
|
||||
- Add "Use Suggested" / "Keep Original" buttons
|
||||
|
||||
- [ ] **Task 2.3**: Create confirmation dialog
|
||||
- Create `URLCorrectionDialog` component
|
||||
- Shows original vs suggested URL comparison
|
||||
- Handles accept/reject/cancel actions
|
||||
- Integrate with repository creation flow
|
||||
|
||||
## Phase 3: Integration & Testing
|
||||
|
||||
- [ ] **Task 3.1**: Wire up frontend to backend
|
||||
- Call parse-url endpoint on URL input change (debounced)
|
||||
- Handle 422 responses from repository creation
|
||||
- Show confirmation dialog when needed
|
||||
|
||||
- [ ] **Task 3.2**: Add error handling
|
||||
- Handle network errors during URL validation
|
||||
- Show clear error messages for unsupported URLs
|
||||
- Handle clone failures gracefully
|
||||
|
||||
- [ ] **Task 3.3**: Run quality gates
|
||||
- Backend: ruff, mypy, pytest
|
||||
- Frontend: typecheck, lint, build
|
||||
|
||||
- [ ] **Task 3.4**: Manual testing
|
||||
- Test with GitHub URLs (tree, blob, pull)
|
||||
- Test with GitLab URLs (-/tree, -/blob)
|
||||
- Test with valid git URLs (should pass through)
|
||||
- Test force_original_url flag
|
||||
Reference in New Issue
Block a user