User Tools

Site Tools


go:stdlib:reading_writing_patterns

Reading/Writing patterns in Go: one-shot vs stream vs buffered

Goal

Help you choose the right approach when reading/writing data in Go:

  • One-shot (read/write all at once)
  • Stream (unbuffered) (read/write piece by piece)
  • Stream (buffered with bufio) (stream + RAM buffer for speed)

1) One-shot (read all / write all)

Meaning: load the whole content into memory (RAM) in one go.

Common functions

  • `os.ReadFile(path)` → `[]byte`
  • `io.ReadAll(r io.Reader)` → `[]byte`
  • `os.WriteFile(path, data, perm)` → write all bytes at once

When to use

  • File/data is small enough to fit comfortably in RAM.
  • You want simplest code.

Example

b, err := os.ReadFile("data.txt")
if err != nil { return err }
// use b...

2) Stream (unbuffered)

Meaning: read/write gradually (chunks). You do NOT load everything into RAM.

Common functions / methods

  • `f.Read(p []byte)` / `f.Write(p []byte)` (methods on `*os.File`, `net.Conn`, etc.)
  • `io.Copy(dst, src)` (stream copy)
  • `io.CopyN(dst, src, n)` (copy exactly n bytes)
  • `io.ReadFull(r, buf)` (fill buf or error)

When to use

  • Large files, network streams, unknown size.
  • You want low memory usage.

Example (read chunk-by-chunk)

buf := make([]byte, 4096)
for {
    n, err := f.Read(buf)
    if n > 0 {
        // process buf[:n]
    }
    if err == io.EOF {
        break
    }
    if err != nil {
        return err
    }
}

3) Stream (buffered with bufio)

Meaning: still stream-based, but `bufio` keeps an in-memory buffer to reduce system calls. It reads/writes larger blocks internally, while you consume/produce smaller pieces.

Common functions

  • `bufio.NewReader®` / `bufio.NewReaderSize(r, size)`
  • `bufio.NewWriter(w)` / `bufio.NewWriterSize(w, size)`
  • Reader helpers: `ReadString('\n')`, `ReadBytes('\n')`, `ReadLine()`
  • Writer helpers: `WriteString(…)`, `Write(…)`, then `Flush()`

When to use

  • Reading lines (text), lots of small reads/writes.
  • Network/file I/O where you read/write frequently in small pieces.

Example (read lines)

rd := bufio.NewReader(f)
for {
    line, err := rd.ReadString('\n')
    if len(line) > 0 {
        // use line
    }
    if err == io.EOF {
        break
    }
    if err != nil {
        return err
    }
}

Example (buffered writer)

wr := bufio.NewWriter(f)
_, _ = wr.WriteString("hello\n")
_ = wr.Flush() // important!

4) Stream token/line scanning (bufio.Scanner)

Meaning: convenient tokenization (often line-by-line), great for simple text parsing.

Common functions

  • `scanner := bufio.NewScanner®`
  • `scanner.Scan()` (loop)
  • `scanner.Text()` / `scanner.Bytes()`
  • `scanner.Split(bufio.ScanLines)` (default) or other split funcs

When to use

  • Simple line-based reading.
  • Quick parsing tasks.

Pitfall

`Scanner` has a default max token size. For very long lines, configure buffer or use `bufio.Reader`.

Quick decision table

Your case Recommended
Small file, want simplest `os.ReadFile` / `os.WriteFile`
Large/unknown size stream `io.Copy` or `Read` loop
Many small reads/writes `bufio.Reader` / `bufio.Writer`
Read line-by-line simply `bufio.Scanner` (or `Reader.ReadString`)

Why bufio is faster (in one sentence)

Because it reduces expensive OS calls by doing fewer, bigger reads/writes under the hood.

Hard words (English)

  • one-shot /ˌwʌn ˈʃɑːt/: làm một phát (đọc/ghi hết)
  • stream /striːm/: luồng dữ liệu
  • chunk /tʃʌŋk/: khối dữ liệu
  • buffer /ˈbʌfər/: bộ đệm (vùng RAM tạm)
  • unbuffered /ʌnˈbʌfərd/: không dùng bộ đệm
  • token /ˈtoʊkən/: đơn vị tách (mảnh dữ liệu)
  • system call /ˈsɪstəm kɔːl/: lời gọi hệ điều hành
  • under the hood /ˈʌndər ðə hʊd/: ở bên trong (cơ chế nội bộ)
go/stdlib/reading_writing_patterns.txt · Last modified: by phong2018