Streams

The Util.Streams package provides several types and operations to allow the composition of input and output streams. Input streams can be chained together so that they traverse the different stream objects when the data is read from them. Similarly, output streams can be chained and the data that is written will traverse the different streams from the first one up to the last one in the chain. During such traversal, the stream object is able to bufferize the data or make transformations on the data.

The Input_Stream interface represents the stream to read data. It only provides a Read procedure. The Output_Stream interface represents the stream to write data. It provides a Write, Flush and Close operation.

To use the packages described here, use the following GNAT project:

with "utilada_sys";

Buffered Streams

The Output_Buffer_Stream and Input_Buffer_Stream implement an output and input stream respectively which manages an output or input buffer. The data is first written to the buffer and when the buffer is full or flushed, it gets written to the target output stream.

The Output_Buffer_Stream must be initialized to indicate the buffer size as well as the target output stream onto which the data will be flushed. For example, a pipe stream could be created and configured to use the buffer as follows:

with Util.Streams.Buffered;
with Util.Streams.Pipes;
...
   Pipe   : aliased Util.Streams.Pipes.Pipe_Stream;
   Buffer : Util.Streams.Buffered.Output_Buffer_Stream;
   ...
      Buffer.Initialize (Output => Pipe'Unchecked_Access,
                         Size => 1024);

In this example, the buffer of 1024 bytes is configured to flush its content to the pipe input stream so that what is written to the buffer will be received as input by the program. The Output_Buffer_Stream provides write operation that deal only with binary data (Stream_Element). To write text, it is best to use the Print_Stream type from the Util.Streams.Texts package as it extends the Output_Buffer_Stream and provides several operations to write character and strings.

The Input_Buffer_Stream must also be initialized to also indicate the buffer size and either an input stream or an input content. When configured, the input stream is used to fill the input stream buffer. The buffer configuration is very similar as the output stream:

with Util.Streams.Buffered;
with Util.Streams.Pipes;
...
   Pipe   : aliased Util.Streams.Pipes.Pipe_Stream;
   Buffer : Util.Streams.Buffered.Input_Buffer_Stream;
   ...
      Buffer.Initialize (Input => Pipe'Unchecked_Access, Size => 1024);

In this case, the buffer of 1024 bytes is filled by reading the pipe stream, and thus getting the program's output.

Texts

The Util.Streams.Texts package implements text oriented input and output streams. The Print_Stream type extends the Output_Buffer_Stream to allow writing text content.

The Reader_Stream type extends the Input_Buffer_Stream and allows to read text content.

File streams

The Util.Streams.Files package provides input and output streams that access files on top of the Ada Stream_IO standard package. The File_Stream implements both the Input_Stream and Output_Stream interfaces. The stream is opened by using the Open or Create procedures.

with Util.Streams.Files;
...
  In_Stream : Util.Streams.Files.File_Stream;
  In_Stream.Open (Mode => Ada.Streams.Stream_IO.In_File, "cert.pem");

Pipes

The Util.Streams.Pipes package defines a pipe stream to or from a process. It allows to launch an external program while getting the program standard output or providing the program standard input. The Pipe_Stream type represents the input or output stream for the external program. This is a portable interface that works on Unix and Windows.

The process is created and launched by the Open operation. The pipe allows to read or write to the process through the Read and Write operation. It is very close to the popen operation provided by the C stdio library. First, create the pipe instance:

with Util.Streams.Pipes;
...
   Pipe : aliased Util.Streams.Pipes.Pipe_Stream;

The pipe instance can be associated with only one process at a time. The process is launched by using the Open command and by specifying the command to execute as well as the pipe redirection mode:

READ to read the process standard output,
WRITE to write the process standard input.

For example to run the ls -l command and read its output, we could run it by using:

Pipe.Open (Command => "ls -l", Mode => Util.Processes.READ);

The Pipe_Stream is not buffered and a buffer can be configured easily by using the Input_Buffer_Stream type and connecting the buffer to the pipe so that it reads the pipe to fill the buffer. The initialization of the buffer is the following:

with Util.Streams.Buffered;
...
   Buffer : Util.Streams.Buffered.Input_Buffer_Stream;
   ...
   Buffer.Initialize (Input => Pipe'Unchecked_Access, Size => 1024);

And to read the process output, one can use the following:

 Content : Ada.Strings.Unbounded.Unbounded_String;
 ...
 Buffer.Read (Into => Content);

The pipe object should be closed when reading or writing to it is finished. By closing the pipe, the caller will wait for the termination of the process. The process exit status can be obtained by using the Get_Exit_Status function.

 Pipe.Close;
 if Pipe.Get_Exit_Status /= 0 then
    Ada.Text_IO.Put_Line ("Command exited with status "
                          & Integer'Image (Pipe.Get_Exit_Status));
 end if;

You will note that the Pipe_Stream is a limited type and thus cannot be copied. When leaving the scope of the Pipe_Stream instance, the application will wait for the process to terminate.

Before opening the pipe, it is possible to have some control on the process that will be created to configure:

The shell that will be used to launch the process,
The process working directory,
Redirect the process output to a file,
Redirect the process error to a file,
Redirect the process input from a file.

All these operations must be made before calling the Open procedure.

Sockets

The Util.Streams.Sockets package defines a socket stream.

Raw files

The Util.Streams.Raw package provides a stream directly on top of file system operations read and write.

Part streams

The Input_Part_Stream is an input stream which decomposes an input stream in several parts separated by well known and fixed boundaries. It can be used to read multipart streams, certificate files, private and public keys and others. The example below shows how to read a file composed of several parts separated by well defined text boundaries.

with Util.Streams.Files;
with Util.Streams.Buffered.Parts;
...
  In_Stream   : aliased Util.Streams.Files.File_Stream;
  Part_Stream : Util.Streams.Buffered.Parts.Input_Part_Stream;

With the above declarations, the Input_Part_Stream is configured to read from the File_Stream by using the Initialize procedure and giving a buffer size. The buffer size must be large enough to hold the largest fixed boundary plus some extra.

Part_Stream.Initialize (Input => In_Stream'Unchecked_Access, Size => 4096);

Once it is configured, the first boundary to stop at is configured by using the Set_Boundary procedure. The example below is intended to extract the certificate from a PEM file. The certificate (encoded in Base64) is enclosed in two different markers. The first boundary is first defined as follows:

Part_Stream.Set_Boundary ("-----BEGIN CERTIFICATE-----" & ASCII.LF);

After calling Set_Boundary, we can start reading the Part_Stream and it will stop when the boundary string is found. If we want to drop content until the first boundary is found, we can loop until the boundary is found. To extract the certificate content, we want to skip everything until the first boundary is found in the stream:

while not Part_Stream.Is_Eob loop
  Part_Stream.Read (Item);
end loop;

Once a boundary is reached, trying to read from the stream will raise the standard Data_Error exception. We can either use Next_Part to prepare and read for a next part with the same boundary or call Set_Boundary with another boundary. To extract the certificate content, we can do:

Part_Stream.Set_Boundary ("-----END CERTIFICATE-----" & ASCII.LF);
Part_Stream.Read (Content);

Encoder Streams

The Util.Streams.Buffered.Encoders is a generic package which implements an encoding or decoding stream through the Transformer interface. The generic package must be instantiated with a transformer type. The stream passes the data to be written to the Transform method of that interface and it makes transformations on the data before being written.

The AES encoding stream is created as follows:

package Encoding is
  new Util.Streams.Buffered.Encoders (Encoder => Util.Encoders.AES.Encoder);

and the AES decoding stream is created with:

package Decoding is
  new Util.Streams.Buffered.Encoders (Encoder => Util.Encoders.AES.Decoder);

The encoding stream instance is declared:

  Encode : Util.Streams.Buffered.Encoders.Encoder_Stream;

The encoding stream manages a buffer that is used to hold the encoded data before it is written to the target stream. The Initialize procedure must be called to indicate the target stream, the size of the buffer and the encoding format to be used.

 Encode.Initialize (Output => File'Access, Size => 4096, Format => "base64");

The encoding stream provides a Produces procedure that reads the encoded stream and write the result in another stream. It also provides a Consumes procedure that encodes a stream by reading its content and write the encoded result to another stream.

Base16 Encoding Streams

The Util.Streams.Base16 package provides streams to encode and decode the stream using Base16.

Base64 Encoding Streams

The Util.Streams.Base64 package provides streams to encode and decode the stream using Base64.

AES Encoding Streams

The Util.Streams.AES package define the Encoding_Stream and Decoding_Stream types to encrypt and decrypt using the AES cipher. Before using these streams, you must use the Set_Key procedure to setup the encryption or decryption key and define the AES encryption mode to be used. The following encryption modes are supported:

AES-ECB
AES-CBC
AES-PCBC
AES-CFB
AES-OFB
AES-CTR

The encryption and decryption keys are represented by the Util.Encoders.Secret_Key limited type. The key cannot be copied, has its content protected and will erase the memory once the instance is deleted. The size of the encryption key defines the AES encryption level to be used:

Use 16 bytes, or Util.Encoders.AES.AES_128_Length for AES-128,
Use 24 bytes, or Util.Encoders.AES.AES_192_Length for AES-192,
Use 32 bytes, or Util.Encoders.AES.AES_256_Length for AES-256.

Other key sizes will raise a pre-condition or constraint error exception. The recommended key size is 32 bytes to use AES-256. The key could be declared as follows:

Key : Util.Encoders.Secret_Key
          (Length => Util.Encoders.AES.AES_256_Length);

The encryption and decryption key are initialized by using the Util.Encoders.Create operations or by using one of the key derivative functions provided by the Util.Encoders.KDF package. A simple string password is created by using:

Password_Key : constant Util.Encoders.Secret_Key
          := Util.Encoders.Create ("mysecret");

Using a password key like this is not the good practice and it may be useful to generate a stronger key by using one of the key derivative function. We will use the PBKDF2 HMAC-SHA256 with 20000 loops (see RFC 8018):

Util.Encoders.KDF.PBKDF2_HMAC_SHA256 (Password => Password_Key,
                                      Salt     => Password_Key,
                                      Counter  => 20000,
                                      Result   => Key);

To write a text, encrypt the content and save the file, we can chain several stream objects together. Because they are chained, the last stream object in the chain must be declared first and the first element of the chain will be declared last. The following declaration is used:

  Out_Stream   : aliased Util.Streams.Files.File_Stream;
  Cipher       : aliased Util.Streams.AES.Encoding_Stream;
  Printer      : Util.Streams.Texts.Print_Stream;

The stream objects are chained together by using their Initialize procedure. The Out_Stream is configured to write on the encrypted.aes file. The Cipher is configured to write in the Out_Stream with a 32Kb buffer. The Printer is configured to write in the Cipher with a 4Kb buffer.

  Out_Stream.Initialize (Mode => Ada.Streams.Stream_IO.In_File,
                         Name => "encrypted.aes");
  Cipher.Initialize (Output => Out_Stream'Unchecked_Access,
                     Size   => 32768);
  Printer.Initialize (Output => Cipher'Unchecked_Access,
                      Size   => 4096);

The last step before using the cipher is to configure the encryption key and modes:

  Cipher.Set_Key (Secret => Key, Mode => Util.Encoders.AES.ECB);

It is now possible to write the text by using the Printer object:

Printer.Write ("Hello world!");