earthkit-data: streams - ECMWF internal earthkit training, March 2026

In this notebook you will see how to:

read GRIB from a stream
iterate through a stream
read parts (bytes ranges)

import earthkit.data as ekd

Loading URL as a stream¶

We read GRIB data from a URL as a stream. With this we can avoid disk usage and can work entirely in memory.

url = "https://sites.ecmwf.int/repository/earthkit/samples/test6.grib"
d = ekd.from_source("url", url, stream=True)
d

The usage of this data is rather limited, but we can convert it into a stream fieldlist and iterate through it field by field. This is a special fieldlist, apart from the iteration no other methods are available, e.g. len() does not work.

fl = d.to_fieldlist()
try:
    len(fl)
except Exception as e:
    print(e)

When we iterate through the stream fieldlist Field objects are created then get deleted when going out of scope. As a result, only one GRIB message is kept in memory at a time.

for f in fl:
    print(f)

When we finished the iteration we consumed all the data from the stream. Another attemp to iterate would yield nothing.

for f in fl:
    print(f)

Using parts¶

We can use parts (byte ranges) for URLs and it also works for streams.

# Only read the bytes for the ranges of 0-240 and 480-960 from the URL
# This contains GRIB message 0, 2 and 3.
# The part specification has the format of (start, length)
d = ekd.from_source(
        "url",
        url,
        parts=[(0, 240), (480, 480)], stream=True)

for f in d.to_fieldlist():
    print(f)

Reading the whole stream into memory¶

If you know that the full stream contents fits into memory you can consume the whole stream a load it into a fieldlist in memory. This fiedlist now has all the usual fieldlist methods/attributes.

# fl is a fieldlist entirely in memory
d = ekd.from_source("url", url, stream=True)
fl = d.to_fieldlist(read_all=True)
fl.ls()

Please use this option carefully!