•1 min read
Using HuggingFace Datasets Offline
How to save a HuggingFace dataset to disk and use it offline
#ml/ai
View Raw (for LLMs)
This is pretty simple, but quite helpful if you're running jobs on a compute node that doesn't have internet access.
On the login node or another machine with internet access, run the following Python code:
import datasets
x = datasets.load_dataset("my_dataset")
x.save_to_disk("./my_dataset_local")Then, if needed, copy the files to the machine running your job. Now, from that offline machine, loading the dataset is simple!
y = datasets.load_from_disk("./hellaswag_local")
y # DatasetDict({...})