Supported Data Formats#

The expected flow is always Ray Dataset -> Kithara Dataset -> Kithara Dataloader.

For example,:

hf_dataset = load_dataset("allenai/c4", "en", split="train", streaming=True)
ray_dataset = ray.data.from_huggingface(hf_dataset)

kithara_dataset = Kithara.TextCompletionDataset(
    ray_dataset,
    tokenizer_handle="hf://google/gemma-2-2b",
    max_seq_len=8192
)
dataloader = Kithara.Dataloader(
    kithara_dataset,
    per_device_batch_size=1
)

Note

You can load files and other data from your local directory, S3, GCS, ABS, and any other data storage platforms supported by Ray Data.

Kithara supports all file formats supported by Ray Dataset. Listed below are some examples using popular dataset types.

HuggingFace Dataset#

Load datasets directly from the Hugging Face Hub. Streaming datasets are supported:

hf_dataset = load_dataset("allenai/c4", "en", split="train", streaming=True)
ray_dataset = ray.data.from_huggingface(hf_dataset)

JSON and JSONL#

Read JSON and JSONL files.

ray_dataset = ray.data.read_json("s3://anonymous@ray-example-data/log.json")

CSV#

Read CSV files.

ray_dataset = ray.data.read_csv("s3://anonymous@ray-example-data/iris.csv")

Python Dictionary#

dataset_items = [{
    "prompt": "What is your name?",
    "answer": "My name is Kithara",
}  for _ in range(1000)]

ray_dataset = ray.data.from_items(dataset_items)

Pandas#

df = pd.DataFrame({
"food": ["spam", "ham", "eggs"],
"price": [9.34, 5.37, 0.94]
})
ray_dataset = ray.data.from_pandas(df)

Lines of Text#

Read text files.

ray_dataset = ray.data.read_text("s3://anonymous@ray-example-data/this.txt")

TFRecords#

Read TFRecords Files

ray_dataset = ray.data.read_tfrecords("s3://anonymous@ray-example-data/iris.tfrecords")

Parquet#

Read Parquet files.

ray_dataset = ray.data.read_parquet("s3://anonymous@ray-example-data/iris.parquet")

Additional Formats#

Kithara supports all Ray Data formats. For more information:

Note: You should handle data cleaning, transformation, shuffling, and splitting using Ray Dataset utilities before passing the data to Kithara Dataset.