Split any iterable into batches
Yield fixed-size lists from a list, generator or file without loading everything into memory.
- Python
- C#
- GreenSnippets
- MIT
- Executed with Python 3.11 (Python tab only)
from itertools import islice
def batched(iterable, size):
if size < 1:
raise ValueError("size must be >= 1")
iterator = iter(iterable)
while batch := list(islice(iterator, size)):
yield batchOutput
list(batched(range(5), 2)) # [[0, 1], [2, 3], [4]]using System;
using System.Collections.Generic;
using System.Linq;
public static class Batching
{
// .NET 6+ also ships Enumerable.Chunk; this version is lazy and works on older targets.
public static IEnumerable<List<T>> Batched<T>(IEnumerable<T> source, int size)
{
if (size < 1) throw new ArgumentOutOfRangeException(nameof(size));
var batch = new List<T>(size);
foreach (var item in source)
{
batch.Add(item);
if (batch.Count == size)
{
yield return batch;
batch = new List<T>(size);
}
}
if (batch.Count > 0) yield return batch;
}
}Output
Batching.Batched(Enumerable.Range(0, 5), 2) // [[0, 1], [2, 3], [4]]How it works
Python 3.12 adds itertools.batched; this version works on older interpreters and returns lists. Because it consumes the iterable lazily, you can batch a huge file or database cursor while holding only one batch in memory.
Related snippets
- Python Tested
Time a block of code with a context manager
Measure elapsed wall-clock time of a with-block using time.perf_counter.
- PHP Tested
Read a large CSV file with a generator
Stream CSV rows as associative arrays using fgetcsv, without loading the whole file into memory.
- JavaScript Tested
Run async tasks with a concurrency limit
Process a large list with at most N promises in flight at once, preserving result order.
- Bash
Run commands in parallel with xargs
Process many files concurrently with xargs -P, safely handling spaces and newlines in names.