Reading/Writing patterns in Go: one-shot vs stream vs buffered
Goal
Help you choose the right approach when reading/writing data in Go:
One-shot (read/write all at once)
Stream (unbuffered) (read/write piece by piece)
Stream (buffered with bufio) (stream + RAM buffer for speed)
1) One-shot (read all / write all)
Meaning: load the whole content into memory (RAM) in one go.
Common functions
`os.ReadFile(path)` → `[]byte`
`io.ReadAll(r io.Reader)` → `[]byte`
`os.WriteFile(path, data, perm)` → write all bytes at once
When to use
Example
b, err := os.ReadFile("data.txt")
if err != nil { return err }
// use b...
2) Stream (unbuffered)
Meaning: read/write gradually (chunks). You do NOT load everything into RAM.
Common functions / methods
`f.Read(p []byte)` / `f.Write(p []byte)` (methods on `*os.File`, `net.Conn`, etc.)
`io.Copy(dst, src)` (stream copy)
`io.CopyN(dst, src, n)` (copy exactly n bytes)
`io.ReadFull(r, buf)` (fill buf or error)
When to use
Large files, network streams, unknown size.
You want low memory usage.
Example (read chunk-by-chunk)
buf := make([]byte, 4096)
for {
n, err := f.Read(buf)
if n > 0 {
// process buf[:n]
}
if err == io.EOF {
break
}
if err != nil {
return err
}
}
3) Stream (buffered with bufio)
Meaning: still stream-based, but `bufio` keeps an in-memory buffer to reduce system calls.
It reads/writes larger blocks internally, while you consume/produce smaller pieces.
Common functions
`bufio.NewReader®` / `bufio.NewReaderSize(r, size)`
`bufio.NewWriter(w)` / `bufio.NewWriterSize(w, size)`
Reader helpers: `ReadString('\n')`, `ReadBytes('\n')`, `ReadLine()`
Writer helpers: `WriteString(…)`, `Write(…)`, then `Flush()`
When to use
Reading lines (text), lots of small reads/writes.
Network/file I/O where you read/write frequently in small pieces.
Example (read lines)
rd := bufio.NewReader(f)
for {
line, err := rd.ReadString('\n')
if len(line) > 0 {
// use line
}
if err == io.EOF {
break
}
if err != nil {
return err
}
}
Example (buffered writer)
wr := bufio.NewWriter(f)
_, _ = wr.WriteString("hello\n")
_ = wr.Flush() // important!
4) Stream token/line scanning (bufio.Scanner)
Meaning: convenient tokenization (often line-by-line), great for simple text parsing.
Common functions
`scanner := bufio.NewScanner®`
`scanner.Scan()` (loop)
`scanner.Text()` / `scanner.Bytes()`
`scanner.Split(bufio.ScanLines)` (default) or other split funcs
When to use
Pitfall
`Scanner` has a default max token size. For very long lines, configure buffer or use `bufio.Reader`.
Quick decision table
| Your case | Recommended |
| Small file, want simplest | `os.ReadFile` / `os.WriteFile` |
| Large/unknown size stream | `io.Copy` or `Read` loop |
| Many small reads/writes | `bufio.Reader` / `bufio.Writer` |
| Read line-by-line simply | `bufio.Scanner` (or `Reader.ReadString`) |
Why bufio is faster (in one sentence)
Because it reduces expensive OS calls by doing fewer, bigger reads/writes under the hood.
Related pages
Hard words (English)
one-shot /ˌwʌn ˈʃɑːt/: làm một phát (đọc/ghi hết)
stream /striːm/: luồng dữ liệu
chunk /tʃʌŋk/: khối dữ liệu
buffer /ˈbʌfər/: bộ đệm (vùng RAM tạm)
unbuffered /ʌnˈbʌfərd/: không dùng bộ đệm
token /ˈtoʊkən/: đơn vị tách (mảnh dữ liệu)
system call /ˈsɪstəm kɔːl/: lời gọi hệ điều hành
under the hood /ˈʌndər ðə hʊd/: ở bên trong (cơ chế nội bộ)