pi-share-hf
https://github.com/badlogic/pi-share-hf- Category
- Developer Tools
- Rank
- No. 1956Tools index
- Pricing
- Open Source
- Type
- TOOL
- Builder
- badlogic
- GitHub
- 252 stars
- Date
About
Collect, review, and upload redacted pi session files to a Hugging Face dataset — share your agent runs for evals or research.
What it does
A local command-line pipeline turns one project’s pi coding history into a maintained public dataset. It tracks prior work, removes known literal secrets, blocks deny-pattern matches and scanner findings, asks a language model to judge project relevance and disclosure risk, then publishes approved changes.
Why it's ranked here
The layered release gate is thoughtful and unusually concrete. Exact-value replacement, TruffleHog scanning, contextual model review, manual rejection, upload previews, and unchanged-session detection address distinct failure modes. The catch is important: its own generated dataset warning describes redaction as best effort, so publication still demands human judgment.
What's good
Any TruffleHog result blocks publication, including unverified and unknown findings. Model approval also requires an affirmative sharing decision, no apparent missed sensitive data, and at least partial project relevance. Review caching accounts for content, context, model settings, deny patterns, prompt version, and chunk size. Upload previews and targeted search support a final human pass.
Tradeoffs
Deterministic removal only catches exact values supplied directly or parsed from sensitive-looking environment variables. The workflow requires pi, TruffleHog, a Hugging Face write token, project context, and an LLM review provider. Preserved images need separate inspection. A mixed session can still expose private or off-topic material despite every automated check.
How to use it well
Use it for periodically publishing traces from one open-source project, especially when repeated runs should process only changed sessions. Supply fresh literal secrets, maintain project-specific deny expressions, provide strong context, preview the uploadable set, search it for private topics, inspect preserved images, and reject doubtful sessions. It does not provide a guarantee of anonymization or replace human release approval.
Technical notes+
README.md defines the incremental workspace, command workflow, dependency setup, review gates, and dataset layout. package.json exposes an ESM npm CLI, ships compiled output plus README.md, and depends on @huggingface/hub. src/index.ts dispatches seven commands after startup checks from src/process.ts. src/collect.ts hashes sources and redaction inputs, reuses matching local or remote state, batches TruffleHog scans, and invokes review. src/redactor.ts recursively replaces literal secrets and optionally removes embedded image data. src/query.ts derives the uploadable set from manifest, rejection, and aggregate review state. src/upload.ts stages changed approved sessions, a manifest, and a generated dataset card before src/hf.ts uploads through the TypeScript Hub client.
Observed
- Primary language
- TypeScript with ECMAScript modules
- Packaging
- npm package with a global command-line executable
- Interface
- CLI commands: init, collect, review, upload, reject, list, and grep
- Runtime dependency
- Uses the @huggingface/hub TypeScript client
- External tools
- Collection requires pi and TruffleHog
- Authentication
- Accepts Hugging Face tokens from environment variables or local token files
- Platform guidance
- Documents TruffleHog setup for macOS, Linux, and Windows
Read from README.md, package.json, src/cli.ts, src/index.ts, src/hf.ts, src/query.ts, src/types.ts, src/colors.ts, src/reject.ts, src/review.ts, src/upload.ts, src/collect.ts, src/process.ts, src/secrets.ts, src/redactor.ts.
What it can do
Collect pi session files
Pi agent session data → Collected session files
Review session files
Pi session files → Reviewed and validated session data
Redact personal information from session files
Raw pi session files → Redacted session files with sensitive data removed
Upload session files to Hugging Face dataset
Redacted pi session files → Published dataset on Hugging Face platform
Share agent runs for evaluation purposes
Pi agent execution data → Publicly accessible evaluation dataset
Share agent runs for research purposes
Pi agent execution data → Research-ready dataset
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.