Files

An instance of class pnetcdf.File is a high-level object representing a netCDF file. The class methods provide a set of Pythonic interfaces to create, read and write a netCDF file. A File instance serves as the root container for dimensions, variables, and attributes. Together they describe the meaning of data and relations among data objects stored in a netCDF file.

class pnetcdf.File

Bases: object

__init__(self, filename, format=None, mode="w", MPI.Comm comm=None, MPI.Info info=None)

The constructor for pnetcdf.File.

Parameters:

filename (str) – Name of the new file.
mode (str) –
Access mode.
- r: Open a file for read-only mode. An error will return if the file does not exist.
- w: Create a new file. If a file with the same name has already existed, the file will be clobbered to zero size.
- x: Create a new file, but return an error if the file has already existed.
- a or r+: append mode. It creates the file if it does not exist.
format (str) –
[Optional] underlying file format. Only relevant when creating a new file.
- NETCDF3_64BIT_OFFSET or NC_64BIT_OFFSET: CDF-2 format
- NETCDF3_64BIT_DATA or NC_64BIT_DATA: CDF-5 format
- NETCDF3_CLASSIC or None defaults to default file format (CDF-1 format)
comm (mpi4py.MPI.Comm or None) – [Optional] MPI communicator to use for file access. None defaults to MPI_COMM_WORLD.
info (mpi4py.MPI.Info or None) – [Optional] MPI info instance to use for file access. None defaults to MPI_INFO_NULL.

Returns:

The created file instance.

Return type:

pnetcdf.File

Example:

A example is available in examples/create_open.py

# create a new file using file clobber mode, i.e. flag "-w"
f = pnetcdf.File(filename = "foo.nc", mode = 'w', comm = MPI.COMM_WORLD, info = None)

# open an existing file for read only
f = pnetcdf.File(filename = "foo.nc", mode = 'r', comm = MPI.COMM_WORLD, info = None)

attach_buff(self, bufsize)

Allow PnetCDF to allocate an internal buffer for accommodating the write requests. This method call is the prerequisite of buffered non-blocking write. A call to File.detach_buff() is required when this buffer is no longer needed.

Parameters:

bufsize (int) – Size of the buffer in the unit of bytes. Can be obtained using numpy.ndarray.nbytes

Example:

A example is available in examples/nonblocking/nonblocking_write.py

# Before calling bput APIs, calculate allocate space needed
bufsize = length * NUM_VARS * np.dtype(np.int32).itemsize

f.attach_buff(bbufsize)

begin_indep(self): Method to leave the file’s current mode from collective data mode and enter into independent data mode. The default mode is collective data mode.

cancel(self, num=None, requests=None, status=None)

This method cancels a list of pending nonblocking requests made by the nonblocking methods, such as Variable.iput_var(), Variable.iget_var(), and Variable.bput_var()

Parameters:

num (int) –
[Optional] Number of requests. It is also the array size of the next two arguments. Alternatively it can be module-level constants:
- pnetcdf.NC_REQ_ALL or None: flush all pending nonblocking requests
- pnetcdf.NC_GET_REQ_ALL: flush all pending nonblocking GET requests
- pnetcdf.NC_PUT_REQ_ALL: flush all pending nonblocking PUT requests
requests (list of int) – [Optional] Integers specifying the nonblocking request IDs that were made earlier.
status (list) – [Optional] List of integers to hold returned error codes from the call, each specifying the status of corresponding nonblocking request. The values can be used in a call to pnetcdf.strerror() to obtain the status messages.

Operational mode:

it can be called in either independent or collective data mode or define mode.

close(self)

Close the opened netCDF file

Example:

A example is available in examples/create_open.py

# create a new file using file clobber mode, i.e. flag "-w"
f = pnetcdf.File(filename = "foo.nc", mode = 'w', comm = MPI.COMM_WORLD, info = None)

f.close()

createDimension(self, dimname, size=-1): Same as File.def_dim()

createVariable(self, varname, datatype, dimensions=(), fill_value=None): Same as File.def_var()

def_dim(self, dimname, size=-1)

Creates a new dimension with the given dimname and size. size must be a positive integer or -1, which stands for “unlimited” (default is -1). The return value is the Dimension class instance describing the new dimension. To determine the current maximum size of the dimension, use the python function len() on the Dimension instance. To determine if a dimension is ‘unlimited’, use the Dimension.isunlimited() method of the Dimension instance.

Parameters:

dimname (str) – Name of the new dimension.
size (int) – [Optional] Size of the new dimension.

Example:

A example is available in examples/put_var.py

dim_t = f.def_dim('time', size = -1)
dim_y = f.def_dim("Y",    size = 100)
dim_x = f.def_dim("X",    size = 200)

# Define a 2D variable of integer type
var = f.def_var("foo", pnetcdf.NC_INT, (dim_y, dim_x))

def_var(self, varname, datatype, dimensions=(), fill_value=None)

Create a new variable with the given parameters.

Parameters:

varname (str) – Name of the new variable.
datatype –
The data type of the new variable. It can be a string that describes a numpy dtype object, a numpy dtype object, or one of PnetCDF data type constant, as shown below.
- pnetcdf.NC_CHAR for text data
- pnetcdf.NC_BYTE for 1-byte signed integer
- pnetcdf.NC_SHORT for 2-byte signed integer
- pnetcdf.NC_INT for 4-byte signed integer
- pnetcdf.NC_FLOAT for 4-byte floating point number
- pnetcdf.NC_DOUBLE for 8-byte real number in double precision
The following are additional data types supported by CDF-5 format.
- pnetcdf.NC_UBYTE for unsigned 1-byte integer
- pnetcdf.NC_USHORT for unsigned 2-byte integer
- pnetcdf.NC_UINT for unsigned 4-byte intege
- pnetcdf.NC_INT64 for signed 8-byte integer
- pnetcdf.NC_UINT64 for unsigned 8-byte integer
dimensions (tuple of str or pnetcdf.Dimension instances) – [Optional] The dimensions of the new variable. Can be either dimension names or dimension class instances. Default is an empty tuple which means the variable is a scalar (and therefore has no dimensions).
fill_value –
[Optional] The fill value of the new variable. Accepted values are shown below.
- None to use the default netCDF fill value for the given data type.
- False to turn off the fill mode.
- If specified with other value, the default netCDF _FillValue (the value that the variable gets filled with before any data is written to it) is replaced with this value.

Returns:

The created variable

Return type:

pnetcdf.Variable

Example:

A example is available in examples/put_var.py

dim_y = f.def_dim("Y", global_ny)
dim_x = f.def_dim("X", global_nx)

# Define a 2D variable of integer type
var = f.def_var("foo", pnetcdf.NC_INT, (dim_y, dim_x))

del_att(self, name)

Delete a netCDF attribute.

Parameters:: name (str) – Name of the attribute
Operational mode:: This method must be called while the file is in define mode.

detach_buff(self)

Detach the write buffer previously attached for buffered non-blocking write

Example:

A example is available in examples/nonblocking/nonblocking_write.py

# Before calling bput APIs, calculate allocate space needed
bufsize = length * NUM_VARS * np.dtype(np.int32).itemsize
f.attach_buff(bbufsize)

# detach the buffer
f.detach_buff()

end_indep(self): Method to leave the file’s current mode from independent data mode and enter into collective data mode. The default mode is collective data mode.

enddef(self): Method to exit the current file’s mode from define mode and place the file in data mode, so variable data can be read or written.

Note

In PnetCDF implementation, a file mode can be in either define or data mode. While in the define mode, metadata can be created and modified, for instance, creating new dimension, variables, and attributes. Variables that have been defined can only be read and written while in the data mode. This requirement is to guarantee the data consistency when running application programs in parallel.

filepath(self, encoding=None)

Method to return the file system path which was used to open/create the Dataset. The path is decoded into a string using sys.getfilesystemencoding() by default.

Parameters:: encoding (str) – [Optional] character encoding of a string attribute (default is utf-8).
Returns:: The file path
Return type:: str

flush(self): Flush data cached in memory to the file system.

get_att(self, name, encoding='utf-8')

Retrieve a netCDF file attribute. Useful when you need to get a netCDF attribute with the same name as one of the reserved python attributes.

Parameters:

name (str) – Name of the attribute.
encoding (str) – [Optional] character encoding of a string attribute (default is utf-8).

Return type:

str or numpy.ndarray

Operational mode:

This method can be called while the file is in either define or data mode (collective or independent).

Example:

A example is available in examples/get_var.py

# Get global attribute named "foo_attr"
str_att = f.get_att("foo_attr")

# Get the variable's attribute named "foo_attr"
str_att = v.foo_attr

inq_buff_size(self): Return the size (in number of bytes) of the attached buffer. This value is the same as the one used in a call to File.attach_buff() earlier. :rtype: int

inq_buff_usage(self)

Return the current usage of the internal attached buffer, set in the call to File.attach_buff().

Return type:: int

inq_get_size(self)

Reports the amount of data that has actually the amount of data that has been actually read from the file since the file is opened/created.

Return type:: int

inq_header_extent(self)

Reports the current file header extent of an opened netCDF file. The amount is the file space allocated for the file header.

Return type:: int

inq_header_size(self)

Reports the current file header size (in bytes) of an opened netCDF file. Note this is the amount of space used by the metadata.

Return type:: int

inq_info(self)

Returns an MPI info instance containing all the file hints used by PnetCDF library.

Return type:: mpi4py.MPI.Info

inq_nreqs(self)

Method to return the number of pending nonblocking requests.

Return type:: int

inq_num_fix_vars(self)

Return the number of fixed-size variables defined for this netCDF file

Return type:: int

inq_num_rec_vars(self)

Returns the number of record variables defined for this netCDF file

Return type:: int

inq_put_size(self)

Reports the amount of data that has actually been written to the file since the file is opened/created.

Return type:: int

inq_recsize(self)

Return the size of record block, sum of individual record sizes (one record each) of all record variables, for this netCDF file.

Return type:: int

inq_striping(self)

Return the file system striping size.

Return type:: int

inq_unlimdim(self)

Return the unlimited dim instance of the file

Returns:: The dimension instance with unlimited size
Return type:: pnetcdf.Dimension

inq_version(self)

Return the file format version, one of the following PnetCDf constants.

pnetcdf.NC_CLASSIC_MODEL indicating the file is CDF-1 format

pnetcdf.NC_64BIT_OFFSET indicating the file is CDF-2 format

pnetcdf.NC_64BIT_DATA indicating the file is CDF-5 format

pnetcdf.NC_NETCDF4 indicating the file is HDF5 format

Return type:: int

ncattrs(self)

Return netCDF attribute names for this File in a list.

Return type:: list

put_att(self, name, value)

Set a global attribute for this file using name,value pair. Especially useful when you need to set a netCDF attribute with the with the same name as one of the reserved python attributes.

Parameters:

name (str) – Name of the new attribute.
value (str, int, float or list of int and float) – Value of the new attribute.

Operational mode:

This method must be called while the file is in define mode.

Example:

A example is available in examples/put_var.py

str_att = "example attribute of type text."
var.foo_attr = str_att

# Equivalently, below uses function call
var.put_att("foo_attr", str_att)

redef(self): Enter define mode, so that dimensions, variables, and attributes can be added or renamed and attributes can be deleted

Note

In PnetCDF implementation, a file mode can be in either define or data mode. While in the define mode, metadata can be created and modified, for instance, creating new dimension, variables, and attributes. Variables that have been defined can only be read and written while in the data mode. This requirement is to guarantee the data consistency when running application programs in parallel.

renameAttribute(self, oldname, newname): Same as File.rename_att()

renameDimension(self, oldname, newname): Same as File.rename_dim()

renameVariable(self, oldname, newname): Same as File.rename_var()

rename_att(self, oldname, newname)

Rename a File attribute named oldname to newname

Parameters:: oldname (str) – Old name of the attribute.
Operational mode:: If the new name is longer than the original name, the netCDF file must be in define mode. Otherwise, the netCDF file can be in either define or data mode.

rename_dim(self, oldname, newname)

Rename a Dimension named oldname to newname

Parameters:

oldname (str) – Old name of the dimension.
newname (str) – New name of the dimension.

Operational mode:

this method is collective subroutine, argument new name must be consistent among all calling processes. If the new name is longer than the old name, then the netCDF file must be in define mode. Otherwise, the netCDF file can be in either define or data mode.

rename_var(self, oldname, newname)

Rename a Variable named oldname to newname

Parameters:

oldname (str) – Old name of the variable.
newname (str) – New name of the variable.

Operational mode:

this method is collective subroutine, argument new name must be consistent among all calling processes. If the new name is longer than the old name, then the netCDF file must be in define mode. Otherwise, the netCDF file can be in either define or data mode.

set_auto_chartostring(self, value)

Call Variable.set_auto_chartostring() for all variables contained in this File. Calling this method only affects existing variables. Variables defined after calling this method will follow the default behaviour.

Parameters:: value (bool) – True or False
Operational mode:: Any

set_fill(self, fillmode)

Sets the fill mode for a netCDF file open for writing and returns the current fill mode. The fill mode can be specified as either NC_FILL or NC_NOFILL. The default mode of PnetCDF is NC_NOFILL. The method call will change the fill mode for all variables defined so far at the time this API is called. In other words, it overwrites the fill mode for all variables previously defined. This method will also change the default fill mode for new variables defined following this call. In PnetCDF, this API only affects non-record variables. In addition, it can only be called while in the define mode. All non-record variables will be filled with fill values (either default or user-defined) at the time File.enddef() is called.

Parameters:

fillmode (int) – pnetcdf.NC_FILL or pnetcdf.NC_NOFILL

Operational mode:

This method is a collective subroutine and must be called in define mode

Example:

A example is available in examples/fill_mode.py

# set the fill mode to NC_FILL for the entire file
old_fillmode = f.set_fill(pnetcdf.NC_FILL)

if old_fillmode == pnetcdf.NC_FILL:
    print("The old fill mode is NC_FILL")
else:
    print("The old fill mode is NC_NOFILL")

# set the fill mode to back to NC_NOFILL for the entire file
f.set_fill(pnetcdf.NC_NOFILL)

sync(self): Writes all buffered data in the File to the disk file.

wait(self, num=None, requests=None, status=None)

Same as File.wait_all() but called in independent data mode

Operational mode:: it is an independent subroutine and must be called while the file is in independent data mode.

wait_all(self, num=None, requests=None, status=None)

This method is a blocking call that wait for the completion of nonblocking I/O requests made by one of more method calls to Variable.iput_var(), Variable.iget_var() and Variable.bput_var()

Parameters:

num (int) –
[Optional] number of requests. It is also the array size of the next two arguments. Alternatively it can be module-level constants:
- pnetcdf.NC_REQ_ALL or None: flush all pending nonblocking requests
- pnetcdf.NC_GET_REQ_ALL: flush all pending nonblocking GET requests
- pnetcdf.NC_PUT_REQ_ALL: flush all pending nonblocking PUT requests
requests (list of int) – [Optional] Integers specifying the nonblocking request IDs returned from the nonblocking requests posted earlier.
status (list) – [Optional] List of integers to hold returned error codes from the call, each specifying the status of corresponding nonblocking request. The values can be used in a call to pnetcdf.strerror() to obtain the error messages.

Operational mode:

it is an collective subroutine and must be called while the file is in collective data mode.

Example:

A example is available in examples/nonblocking/nonblocking_write.py

# Write one variable at a time, using iput APIs
reqs = []
for i in range(NUM_VARS):
    req_id = vars[i].iput_var(buf[i], start = start, count = count)
    reqs.append(req_id)

# commit posted nonblocking requests
req_errs = [None] * NUM_VARS
f.wait_all(NUM_VARS, reqs, req_errs)

Read-only python fields of class pnetcdf.File

The following class fields are read-only and should not be modified by the user.

dimensions

The dimensions dictionary maps the names of dimensions defined in this file as an instance of the pnetcdf.Dimension.

Type: dict

variables

The variables dictionary maps the names of variables defined in this file as an instance of the pnetcdf.Variable.

Type: dict

file_format

The file format of the netCDF file. Possible values are one of the following strings. “CLASSIC”, “CDF2”, “64BIT_OFFSET”, “64BIT”, “CDF5”, “64BIT_DATA”, “NETCDF4” and “BP”.

Type: str