Hugging Face, both directions

The scripts are in the repository, not in the installed package. Clone it, then cd recipes/05-export/hugging-face before running the commands below. Browse this recipe on GitHub.

Push a graded dataset or a LoRA adapter to a Hub repo you own with your own token; through the platform, pull any Hub split onto your account and read its numbers before you train on it. Two routes, and this page says which is which:

route	call	needs
local, your token	`wai.export(rows, "train.jsonl", format="trl", push_to="me/my-set")`	`HF_TOKEN` or `hf auth login`; `pip install 'whileai[hf]'`
local, your token	`wai.hub.push("out/adapter", "me/my-lora")`	the same; a directory holding `adapter_config.json` becomes a model repo
platform	`wai.hf_publish("ds_...")`, `wai.import_hf("ns/name")`, `wai.hf_publish_run("run_...")`	`WHILEAI_API_KEY` and a Hugging Face account connected on the platform

With your own token

Nothing here calls the platform. huggingface_hub does the upload and authenticates the way every HF library does: token=, else HF_TOKEN, else the login hf auth login cached on the machine. Repos are private until you say private=False: a training set or a checkpoint is not a release until you say so.

import whileai as wai

report = wai.export(rows, "train.jsonl", format="trl", push_to="me/my-set")
report["hub"]["url"]  # https://huggingface.co/datasets/me/my-set
wai.hub.push("out/adapter", "me/my-lora")  # a LoRA directory -> huggingface.co/me/my-lora
wai.hub.push(rows, "me/my-set", private=False)  # rows -> train.jsonl in a public dataset repo

Read them back with load_dataset("me/my-set", split="train") and PeftModel.from_pretrained(base, "me/my-lora"). The format="trl" file carries no loss_mask; the export report’s mask_mode says what TRL will train on (every token, or the last assistant turn for mask_mode="final").

Through the platform

Pushing through the platform is a platform feature: it moves a set that already lives on your account (ds_...) or a hosted run’s adapter (run_...) through the Hugging Face account connected on the website, and the SDK never sees that Hub token. One script, three calls. You need WHILEAI_API_KEY and the connected account (the import half of the script works on a public repo without the connection); a minute end to end.

call	what moves	where it lands
`wai.hf_publish("ds_...")`	your rows	`huggingface.co/datasets/<you>/<repo>`, one split per purpose, commit tagged `zp-<dataset id>`
`wai.import_hf("ns/name", split=...)`	any Hub split	a dataset on your account, profiled (pass rate, support, mixed prompts)
`wai.hf_publish_run("run_...")`	a run’s LoRA adapter	`huggingface.co/<you>/<repo>` with a model card, private by default

Versions are commits. Every push is tagged with the While id it came from, so load_dataset(repo, split, revision="zp-ds_...") loads exactly that push, and whileai.json in the repo maps each split to its dataset with history. Pushing a new cut into the same split replaces the old parts and the commit message carries the delta.

Run it

Connect your Hugging Face account once, on any dataset page at https://withwhile.com/platform/datasets (the platform holds the token, the SDK never sees it). Then:

uv add whileai
whileai login                                        # or WHILEAI_API_KEY
cd recipes/05-export/hugging-face
python roundtrip.py                                    # import a public split, profile it, print the numbers
python roundtrip.py --push ds_0123 --repo my-set       # also push one of your sets and print the tag
python roundtrip.py --push-run run_0123 --repo my-lora # also push a finished run's adapter as a model repo

--repo-in ns/name --split train imports a different split; --keep leaves the imported set on your account; --private makes the pushed dataset repo private. Adapter repos are always pushed private: a checkpoint is not a release. Output for the import half:

imported cornell-movie-review-data/rotten_tomatoes:test -> ds_... (1066 rows)
  rows 1066 · prompts 1 · graded 0 · pass – · support –
  (no reward field: profile it after grading, or use it as an eval set)

The imported set is deleted at the end unless you pass --keep. Every call in roundtrip.py is a request to the platform API (WHILEAI_API_URL, default https://api.withwhile.com). That is how the test suite exercises this script offline: it points WHILEAI_API_URL at a local stub and checks the requests the script makes. See tests/recipes/test_example_hugging_face.py. The local route above is the one place the SDK talks to the Hub itself, and it does so with your token.

Get started

Guides

Concepts

Recipes

Reference

Hugging Face, both directions

With your own token

Through the platform

Run it

​With your own token

​Through the platform

​Run it

With your own token

Through the platform

Run it