From the course: Fine-Tune Your LLMs

Unlock the full course today

Join today to access over 23,000 courses taught by industry experts.

Prepare data for fine-tuning

Prepare data for fine-tuning

From the course: Fine-Tune Your LLMs

Start my 1-month free trial Buy for my team

Prepare data for fine-tuning

“

- [Instructor] There are only a few more steps to complete before fine tuning. The next step is to convert the JSON file to the final format needed to train the model, in this case, JSONL. What we're looking at here is a sample file. It is the output that JSONL file. JSONL stands for JSON Lines, and it is very similar to JSON. It simply consists of several lines where each line is a valid JSON object separated by a new line character. This format makes it easier to process because it doesn't require custom parsers. The file is easily processed line by line. Just read a line and parse it as JSON. Let's navigate to code spaces to review the Python code that converts JSON to JSONL. Let's scroll down and find the JSON to JSONL function. It's in this helper functions section and here it is right here, json_to_jsonl. The function first loads the JSON file as shown here, and then it returns the JSON object as a dictionary. Lastly, it produces JSONL from JSON by writing an output file. There…

Contents